Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 40 ยท Deployment and Pitfalls

Pitfalls: Leakage, Extrapolation and Reproducibility

Avoid the most expensive boosting mistakes: target leakage and time-split errors, trees that cannot extrapolate, drift between training and serving, and results that change between runs.

Most failures of gradient-boosted models in production are not about tuning. They are about the data: a feature that secretly contains the answer, a trend the trees cannot continue, or a pipeline that is not reproducible. This lesson walks through the pitfalls that cost teams the most time, each with a small experiment you can run.

Pitfall 1: leakage

Leakage means the model sees information at training time that will not exist at prediction time. Validation scores then look wonderful and production results disappoint. Boosted trees are very good at finding leaks, so they make the problem easy to hit.

Typical sources:

  • A column derived from the target, such as "refund issued" when predicting "will the customer cancel".
  • Identifiers and row order: an ID or timestamp that correlates with the label because of how the data was collected.
  • Future information: features computed over a window that extends past the prediction moment (a 30-day average that includes the next two weeks).
  • Preprocessing fitted on all data: scaling, target encoding or imputation computed before the train/validation split.
  • Duplicates and near-duplicates that appear in both train and validation.
Python A leaky feature inflates validation, and importance shows it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In one run the honest model had a validation AUC of 0.68 and the leaky one 1.000, with the leaky column on top of the importance ranking. Two warning signs are visible: a score that is too good for the problem and a single feature that dominates. When you see either, ask "would I have this value at the moment of prediction?" before celebrating.

Time matters: split by time, not at random

If rows are ordered in time (transactions, sales, sensor data), a random split lets the model train on the future and predict the past. Use a time-based split or TimeSeriesSplit: train on everything before a cutoff, validate after it. The validation score will usually be worse than with a random split, and it will be the honest one.

flowchart LR
  A["Past data<br/>train"] --> B["Cutoff date"]
  B --> C["Future data<br/>validation / test"]
  D["Random split"] -.->|"mixes future into training"| A

Validate the way you will use the model: trained on the past, predicting the future.

Pitfall 2: trees cannot extrapolate

A tree predicts a constant inside each leaf, and the leaves cover only the range of values seen in training. Beyond the largest training value, the prediction stays flat at the last leaf, even if the relationship is a clear upward trend. Linear models extrapolate, trees do not.

Python Trees stay flat outside the training range

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For x beyond 10, the XGBoost prediction stays at about 29.6 while the true value keeps growing (36, 45, 60), and the linear model follows the trend. Consequences and remedies:

  • Be careful with trending targets (prices, demand with growth). Model the change or a ratio (for example y / baseline or y - last_value) instead of the level, or fit a trend with a linear model and let the booster learn the residuals.
  • In LightGBM, linear_tree=True fits a linear model inside each leaf, which can extrapolate (use it with care and check validation).
  • Watch for drift: features whose values move outside the training range in production silently lose accuracy.

Pitfall 3: training/serving skew

The model is only as good as the match between training features and live features. Common causes: a feature computed with different code in the batch job and in the online service, categories that appear in production but not in training, units or timezones that changed, and missing values encoded differently (NaN in training, -1 at serving). Reduce the risk by sharing one feature function between training and serving, logging the live feature values, and monitoring their distributions against the training data: mean, share of missing values and range for numeric columns, category frequencies for categorical ones.

Pitfall 4: results that change between runs

Boosting is deterministic given the same data, parameters and seed, but several things break that:

  • Randomness without a fixed seed: subsample, colsample_* and bagging use random numbers. Set random_state (and seed in native APIs).
  • Threading: floating-point sums in parallel can differ slightly with the thread count, so results can vary in the last digits between machines. LightGBM offers deterministic=True.
  • Library and hardware versions: GPU and CPU histograms differ slightly; different versions may change defaults.
  • Unordered data: a shuffled input order can change a split between two equal candidates.
Python Same seed, same model; different seed, different model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two lessons follow. Fixing the seed makes a run repeatable. And the seed-to-seed spread (about 0.008 of AUC here, from 0.8995 to 0.9074) is the noise floor: do not celebrate a tuning "improvement" smaller than that spread. For a stable final model, average several models trained with different seeds.

Pitfall 5: calibration and drift over time

Boosted probabilities are often well-ordered but not well-calibrated, especially after class weighting (Class Imbalance). Check a reliability curve and recalibrate (CalibratedClassifierCV with isotonic or sigmoid method) on a separate calibration set. Retrain on a schedule or when monitored metrics (score distribution, positive rate, feature drift, and delayed true labels) move.

A pre-launch checklist

  1. Could every feature be known at prediction time?
  2. Is the split by time (or by group, such as customer) whenever rows are related?
  3. Is preprocessing fitted on the training part only, inside a pipeline?
  4. Are seeds fixed, library versions pinned, and the seed-to-seed spread measured?
  5. Are inputs outside the training range, new categories and missing values handled?
  6. Is the threshold, calibration and monitoring plan stored with the model?

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A model achieves 0.99 AUC on validation for a hard problem, and one feature dominates the importance ranking. What should you suspect first?

    A score that is too good for the problem plus one dominating feature suggests that feature contains information from the future or from the target.
  2. 2. Why do tree ensembles predict a flat value beyond the training range?

    Leaves output constants; there is no slope, so the last leaf value repeats for every larger input.
  3. 3. How should you validate a model that will predict next month's sales?

    A time-based split mirrors real usage and prevents the model from learning from the future.
  4. 4. Your tuning changed validation AUC by +0.001, and the seed-to-seed spread is 0.004. What do you conclude?

    Differences smaller than the random-seed spread cannot be distinguished from luck.

Summary

  • Leakage gives unrealistic scores; check feature availability, split by time or group, and fit preprocessing inside the training fold.
  • Trees do not extrapolate: model changes or ratios for trending targets, and monitor features that leave the training range.
  • Avoid training/serving skew with shared feature code and monitoring.
  • Fix seeds, pin versions, and measure the seed-to-seed noise before trusting small gains.
  • Calibrate and monitor, and retrain when the data drifts.

Next: Project: Fraud Detection