Skip to content
Tutorials Center
My learning

Topics

Lesson 85 of 111 ยท Beyond the Basics

Common Machine Learning Mistakes

Learn the mistakes that sink real projects: data leakage, wrong metrics, tuning on the test set, missing baselines, drift over time, tiny datasets and unseeded runs.

Most failed machine learning projects do not fail because someone picked the wrong algorithm. They fail because the evaluation lied. A model looks brilliant in the notebook and disappoints in production, and the reason is almost always one of a handful of classic mistakes: information leaking from the answer into the features, a metric that rewards the wrong behavior, a test set used so often that it became training data, or a world that changed after the model was built.

This lesson gives each mistake a short, runnable demonstration so you recognize it when it bites. Every example uses synthetic data with fixed seeds, so you can see what the mistake does to the numbers. Treat the lesson as a pre-flight checklist: skim it again before you report results to anyone.

Seven ways to fool yourself

Leakage The answer sneaks into the features.
Wrong metric Accuracy on rare events.
Test-set tuning Choosing by the exam you will be graded on.
No baseline Nothing to compare against.
1 / 2

Mistake 1: data leakage

Leakage means the model sees information during training that it will not have when making real predictions. The score then measures something that cannot be repeated in production. It comes in two main forms: target leakage (a feature that is secretly a consequence of the label, such as "account closed" in a churn model) and preprocessing leakage (a transformation fitted on data that includes the test rows). Nested Cross-Validation and Data Leakage goes deeper; here is the most striking demonstration.

We build data with no signal at all: 100 rows, 1,000 random features and random labels. Any honest model should score about 50%. Now select "the 20 best features" and then cross-validate. If the selection is done on all rows first, the labels have already influenced the features the model sees.

Python Feature selection before cross-validation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The leaky version reports a score far above chance on data where nothing can be learned. The only difference is when the labels were allowed to influence the features. The cure is structural: put every step that learns something from data (scalers, imputers, encoders, selectors) inside a Pipeline, and cross-validate the whole pipeline.

Ask of every feature: "Would I know this value at the moment I need the prediction?" If the answer is no (a refund issued after the purchase, a diagnosis code entered after the visit), it is target leakage, however good it makes the score look. A suspiciously perfect feature importance or a score near 100% is a leak until proven otherwise.

Mistake 2: the wrong metric

A metric is a definition of success. Pick one that does not match the goal and the model will optimize the wrong thing. The classic case is accuracy on a rare event. If 1% of transactions are fraud, a model that always says "not fraud" is 99% accurate and catches nothing.

Python 98% accuracy, zero fraud caught

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Accuracy (98.4% against 99.1%) barely separates the two models, while recall and average precision show a large difference: the do-nothing model finds none of the fraud, and even the forest finds fewer than half. The same warning applies to regression: Rยฒ can look excellent on a trending series while the errors are useless, and MAPE explodes when true values are near zero. Choose the metric from the cost of each kind of error (see Classification Metrics and Regression Metrics), decide it before modeling, and report more than one number.

Mistake 3: tuning on the test set

The test set is meant to be an exam taken once. If you look at its score, change something, and look again, over and over, you are fitting the model to the test set one decision at a time. The reported score is then optimistic, and the effect gets stronger the more candidates you try and the smaller the test set is.

To see it, we try 300 candidate models that are all equally mediocre and pick the one that happens to do best on a small test set. Then we grade that winner on a large fresh set.

Python The winner's curse

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The reported number for the winner (0.65) is far higher than what it achieves on fresh data (0.55), which is barely above what the average candidate achieves there (0.51). Selection rewarded luck on 100 rows. The remedies are the ones from earlier lessons: tune with cross-validation on the training data, use nested cross-validation when you must report the tuned score, and touch the final test set once.

Mistake 4: no baseline

"The model reaches an Rยฒ of 0.9" means nothing until you know what a trivial approach scores. A baseline is the cheapest sensible predictor: the mean, the majority class, last period's value, or a short rule written by a domain expert. If a complex model cannot clearly beat it, the complexity is not buying anything. Baselines and Dummy Models covers this in depth.

Time series make the point sharply, because the "yesterday" baseline is hard to beat. The next example also exposes the next mistake, so read both parts.

Mistake 5: ignoring that the world changes

Standard train/test splits assume future data looks like past data. Real data is non-stationary: prices inflate, customers change, sensors age. A random split mixes old and new rows, so the model gets to peek at "the future" during training and scores well. The honest test is to train on the past and evaluate on what comes after.

Here the relationship between the feature and the target changes halfway through the series. We compare a random split, a time-ordered split, and two baselines.

Python Random split versus time split under drift

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The random split makes the forest look very accurate, because for every test row there are near-identical neighbors in time inside the training set. When the model must predict the future, it keeps applying the old rule and the error roughly triples, ending up twice as large as the "previous value" baseline (1.60), which needs no training at all. It still beats the plain training mean (12.09), but a model that loses to "repeat the last value" is not ready to ship. The model is not broken: the world moved.

Practical defenses: use time-ordered splits for anything with a time dimension (see Cross-Validation for Groups and Time), monitor drift after launch (see Saving Models and Deploying), and plan for periodic retraining.

Mistake 6: trusting small data

With few rows, any single train/test split is mostly luck. Which rows land in the test set changes the score noticeably, so a result like "86% accuracy" on 40 test rows carries a margin of error of many points. Report the spread, not a single draw.

The same example shows the next mistake: results you cannot reproduce. A random seed (random_state in scikit-learn, default_rng(seed) in NumPy) fixes the pseudo-random choices so a run can be repeated exactly.

Python How much does one lucky split matter?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The 30 single splits scatter over a wide range, even though nothing about the data or the model changed. A researcher who tried a few seeds and kept the best one would be reporting noise. Repeated cross-validation gives a more stable estimate and, just as importantly, an honest error bar.

Mistake 7: results you cannot reproduce

If you cannot re-create last week's number, you cannot debug it, compare against it or defend it. A reproducibility checklist for every project:

  • Set seeds everywhere: random_state on splitters and estimators, np.random.default_rng(seed) for your own randomness. A seed does not make a model better, it makes a run repeatable, so check results across several seeds.
  • Record versions: pin library versions and save them with the model.
  • Version the data: keep a copy or a hash of the exact dataset and the query that produced it.
  • Keep one script, run top to bottom. A notebook run out of order is a common source of unrepeatable numbers.
  • Log every experiment: parameters, metrics and the code version, so you can answer "what did we try?"

A pre-flight checklist

Before you trust a score, ask:

  1. Does any feature use information from after the prediction moment, and is every learned step inside the pipeline?
  2. Does the metric reflect the real cost of mistakes, and did I look at more than one number?
  3. Has the test set been used only once, at the very end?
  4. How does a trivial baseline score, and by how much do I beat it?
  5. Would a time-ordered split give the same answer?
  6. How large is the spread across splits and seeds?
  7. Can someone else re-run this and get the same result?

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. You select the 20 most correlated features using all rows, then cross-validate. Why is the score unreliable?

    Selection used the labels of every row, including those later used for validation. Put the selector in a Pipeline so it is refit inside each training fold.
  2. 2. Why is 99% accuracy not impressive for fraud detection where 1% of cases are fraud?

    With rare events the majority class dominates accuracy. Recall, precision and average precision show whether the rare cases are actually found.
  3. 3. You try hundreds of model variants and report the best test-set score. What is the likely result on new data?

    Picking the maximum of many noisy estimates is optimistic. Tune with cross-validation and use the test set once.
  4. 4. Which evaluation best matches a model that will predict next month from this month's data?

    A time-ordered split mimics deployment and reveals problems caused by drift that a random split hides.
  5. 5. With only 80 rows, the accuracy changes a lot between random splits. What should you report?

    On small datasets a single split mostly measures luck. Repeated cross-validation gives an average with an error bar.

Summary

  • Leakage lets the answer reach the features: keep every learned preprocessing step inside a pipeline and exclude anything not known at prediction time.
  • Choose the metric from the costs of errors before modeling; accuracy misleads on rare events.
  • The test set is for one final check. Selecting the best of many candidates on it gives an optimistic score.
  • Always compare with a trivial baseline before celebrating a model.
  • Data drifts: for time-dependent problems, evaluate on a later period than you train on.
  • Small datasets give noisy scores; use repeated cross-validation and report the spread.
  • Fix seeds, record versions and log experiments so results can be reproduced.

Next up: apply all of this in a full project, Project: Predicting House Prices.