Skip to content
Tutorials Center
My learning

Topics

Lesson 83 of 111 ยท Beyond the Basics

AutoML Explained

See what AutoML automates, build a mini AutoML loop with random search over pipelines plus an ensemble, and learn when to trust or avoid automated tools.

Much of the work of a machine learning project follows a pattern: clean the data, encode and scale it, try several model families, tune their hyperparameters, compare with cross-validation and perhaps combine the best few. Automated machine learning (AutoML) turns that pattern into a program that searches for a good pipeline for you. You give it a dataset, a target and a time budget, and it returns a model.

AutoML does not replace understanding. It is a fast way to get a strong baseline, to find out which model families suit your data, and to save routine effort. This lesson explains what it automates, builds a small AutoML loop in about 60 lines so that nothing is magic, and discusses the risks.

What AutoML can automate

  • Data preparation: imputing missing values, encoding categories, scaling, dropping constant columns.
  • Feature engineering: generating interactions, polynomial and date features, and selecting among them.
  • Model selection: trying linear models, trees, forests, boosting, nearest neighbors and more.
  • Hyperparameter optimization: random search, Bayesian optimization (hyperparameter search) or evolutionary search.
  • Ensembling: combining the best models by voting or stacking, which is where AutoML systems often gain the last points.
  • Evaluation and reporting: cross-validated scores, leaderboards, sometimes explanations.

Known tools include open-source libraries (auto-sklearn, TPOT, FLAML, H2O AutoML, AutoGluon) and managed cloud services (Google Vertex AI, Azure Automated ML, Amazon SageMaker Autopilot). They differ in what they search and how much they show. They are not available in the in-browser runner, so below we build the core idea from scikit-learn parts.

A mini AutoML loop

The data has eight numeric columns (some with missing values), one categorical column (city) and a binary target. The loop does what an AutoML system does at its heart:

  1. Define a search space of complete pipelines: preprocessing plus a model family plus random hyperparameters.
  2. Sample a configuration, evaluate it with stratified cross-validation on the training data only.
  3. Rank the results, and
  4. Ensemble the best three with soft voting.

Finally the test set, which the search never saw, tells us how well the whole procedure did, including a plain logistic regression baseline for comparison.

Python Random search over pipelines and a voting ensemble

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Fourteen random configurations are enough to find that this data prefers flexible models (the three best pipelines are nearest-neighbor models at 0.91 to 0.92 cross-validated AUC, with boosting and forests close behind at about 0.90) over the plain logistic regression baseline, which reaches only 0.82 on the test set because it cannot represent the interaction between the city and n0. The ensemble of the best three adds another point on the test set (about 0.93 against 0.92). That is the whole recipe, and the real tools add smarter search (Bayesian optimization learns where good configurations lie), larger spaces, time limits, early stopping and stacking with several layers.

Strengths and limits

Where AutoML helps

  • A strong baseline in minutes, which tells you how much a hand-built model has to improve.
  • Exploring which model families suit the data.
  • Teams with little ML experience, or many similar problems to solve quickly.
  • Removing human bias from the model-selection step.

Where it does not

  • It cannot fix the problem definition. If the target is wrong or the question is badly framed, AutoML produces a beautifully tuned answer to the wrong question.
  • Leakage goes unnoticed. A feature that contains the answer will be used with enthusiasm. Check the importances of the winner (feature importance, leakage).
  • Overfitting the validation set. The more configurations you try, the more likely the best one looks good by luck. This is why the test set stayed untouched above, and why serious tools use a separate holdout or nested validation.
  • Black-box models. The winner may be a large ensemble that is hard to explain, slow to serve, or banned by regulation. Constrain the search to models you can deploy.
  • Cost. Large searches burn compute. Set a budget.
  • Domain knowledge still wins. A feature designed with an expert often beats a thousand automatically generated ones.

A sensible way to use it

  1. Define the problem, the split and the metric yourself (workflow).
  2. Establish baselines (baselines).
  3. Run AutoML on the training data with a time budget.
  4. Inspect what it chose and why: the leaderboard, the importances, the errors.
  5. Evaluate the final model once on the untouched test set and check slices and calibration.
  6. Deploy it with the same discipline as any model: tracking, tests and monitoring (CI/CD).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which of these does a typical AutoML system automate?

    AutoML searches the space of pipelines and models. Problem framing, data collection and the decision to deploy remain human tasks.
  2. 2. In the mini AutoML example, why was the test set not used during the search?

    Selecting among many candidates is a fitting step. The test set must stay untouched to give an honest estimate.
  3. 3. What is a major danger of running AutoML on a dataset that contains a leaky feature?

    AutoML maximizes the validation score, so it will happily use any feature that contains the answer.
  4. 4. Why does the plain logistic regression baseline do worse than the AutoML winners in the example?

    The planted interaction is non-linear in the features, so flexible models fit it better.

Summary

  • AutoML automates preprocessing, model and hyperparameter search, and ensembling, and gives you a strong baseline quickly.
  • At its heart it samples pipelines, scores them with cross-validation on training data, ranks them and often ensembles the best.
  • It does not frame the problem, find leaks or guarantee that a model can be deployed, explained or trusted.
  • Keep an untouched test set, inspect the winning model, and apply the same deployment discipline as for any other model.

Next, put several ideas together in a project: Credit Card Fraud Detection.