Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 40 ยท Getting Started

Splits, Baselines and Metrics

Evaluate XGBoost and LightGBM honestly: train, validation and test splits, simple baselines to beat, the right metric for the task, time-based splits and how to spot leakage.

A boosted model can fit almost anything, including noise. That power makes honest evaluation the most important skill in this course. Before tuning a single parameter you need three things: a split that mimics the future, a baseline that tells you what "good" means, and a metric that matches the business question.

Three sets, three jobs

  • Training set: the rows the model learns from.
  • Validation set: rows used during development to compare settings and to stop training early. You look at it many times, so it slowly becomes optimistic.
  • Test set: touched once, at the end, to report the final score.

With small data, replace the single validation set with cross-validation (you will use it with xgb.cv and scikit-learn later). Keep the test set aside either way.

flowchart LR
  D["All labeled data"] --> TR["Train 60%"]
  D --> VA["Validation 20%"]
  D --> TE["Test 20% (use once)"]
  TR --> M["Fit models"]
  VA --> T["Pick settings, early stop"]
  TE --> F["Final honest score"]

The test set stays locked until the very end.

Always start with a baseline

A baseline is the cheapest sensible predictor. If your boosted model cannot beat it clearly, something is wrong (or the problem has no signal).

  • Classification: predict the most frequent class, or the class prior.
  • Regression: predict the mean or median of the training target.
  • A linear model (logistic or linear regression) is the "serious" baseline.
Python Baselines versus boosting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the do-nothing baseline already reaches about 63% accuracy because the classes are imbalanced. That is why accuracy alone can fool you, and why we also print AUC. On this easy dataset a plain logistic regression is also very strong; boosting shines more when relationships are non-linear and interactions matter.

Pick a metric that fits the question

Task Common metrics Notes
Binary classification ROC AUC, log loss, average precision, F1 Use log loss when probabilities matter; average precision for rare positives
Multiclass multi-logloss, accuracy, macro F1 Macro F1 treats classes equally
Regression RMSE, MAE, R squared MAE is robust to outliers; RMSE punishes big misses
Ranking NDCG, MAP Evaluated per query group

The libraries train on a loss (the objective) but can report any evaluation metric during training. They need not be the same: you can train on log loss and monitor AUC.

Python Regression metrics on diabetes data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The baseline R squared is near zero by definition (it predicts the mean). A good model raises it; on this small, noisy dataset a gain to roughly 0.4 is realistic, which is a useful reminder that not every problem has an easy 99%.

Time and groups: split the way the future will arrive

A random split assumes rows are independent. Often they are not.

  • Time series or anything with a date: train on the past, validate on the later period. A random split lets the model peek at the future and overstates quality.
  • Repeated entities (several rows per customer, patient or device): use a group split (GroupKFold) so one entity is never in both train and validation.
  • Rare classes: use stratified splits so each part keeps the class share.
Python Random split versus time split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The time split error is much larger. It is the honest number: the model never saw the later, higher values, and trees cannot extrapolate a trend (this is the limitation from the first lesson). The random split flatters the model because neighbors in time sit on both sides.

Spotting leakage

Leakage is when information that will not exist at prediction time sneaks into the features. Signs: an unrealistically high score (AUC 0.999 on a hard problem), a single feature with enormous importance, or a big drop once deployed. Fixes: drop columns generated after the outcome, fit encoders and imputers inside the training fold only, and split before any preprocessing that looks at the target.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the test set for?

    Using the test set repeatedly to decide things makes its score optimistic, so it is touched once at the end.
  2. 2. Why is a baseline useful?

    A model is only impressive relative to a simple reference such as the majority class or the mean.
  3. 3. For data with a date column and future prediction, which split is best?

    That mirrors deployment, where the model sees only the past.
  4. 4. Which is a classic sign of target leakage?

    A feature that encodes the outcome gives the model the answer, and the score collapses once it is deployed.

Summary

  • Split into train, validation and test; touch the test set once.
  • Beat a baseline (majority class, mean, linear model) before celebrating.
  • Choose a metric that matches the question; accuracy can mislead on imbalanced data.
  • Split by time or by group when rows are not independent.
  • Hunt for leakage whenever a score looks too good.

Next: First Models with XGBoost and LightGBM