Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 40 ยท Imbalanced Data and Custom Losses

Custom Evaluation Metrics

Write a custom evaluation metric for XGBoost and LightGBM, make early stopping watch the number you care about, and learn the differences in signatures and the higher-is-better flag.

The objective is what the model optimizes during training. The evaluation metric is what you watch to decide when to stop and which model is best. They often differ: you train on log loss because it is smooth and differentiable, but you care about PR-AUC, a cost, or the share of cases found in the top 10%. A custom evaluation metric lets early stopping and model comparison use the number that matters.

Signatures

XGBoost (native xgb.train) LightGBM (sklearn API)
Function arguments (predt, dmatrix), labels via dmatrix.get_label() (y_true, y_pred)
Return value (name, value) (name, value, is_higher_better)
Where it is given custom_metric=func in xgb.train eval_metric=func in fit
Direction maximize=True or False in xgb.train decided by is_higher_better

Two details deserve attention. First, with a built-in objective such as binary:logistic or binary, the prediction your function receives is already the probability. (With a custom objective it is the raw margin, as in the previous lesson.) Second, XGBoost needs the direction stated explicitly through maximize; getting it wrong makes early stopping pick the worst round. The scikit-learn wrapper of XGBoost also accepts a callable eval_metric, but its exact behavior has changed between releases, so this lesson uses the stable native API for XGBoost.

A business-cost metric in XGBoost

Suppose each missed positive costs 10 units and each false alarm costs 1, and you plan to use a threshold of 0.3. The metric is the average cost per row on the validation set, lower is better:

Python Early stopping on a custom cost metric (XGBoost)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In one run the log loss rule stopped at round 91 with a validation cost of 0.354 and the cost rule at round 51 with 0.347. The two stopping rules choose different numbers of trees: log loss prefers a smoother, calibrated model, while the cost metric stops where the decisions at the 0.3 threshold are cheapest. Both are legitimate; the important point is that you choose on purpose. The gap here is small, and the cost metric is also the noisier of the two, so treat a difference like this as a tie unless it repeats across several seeds.

A custom metric computed on the validation set and also used for early stopping makes the reported value slightly optimistic. Report final numbers on a separate test set.

A custom metric in LightGBM

LightGBM wants a name, a value and the direction. The function below measures recall in the top 10% of scores, a metric many review teams care about. Higher is better:

Python Recall at top 10 percent as an early-stopping metric (LightGBM)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In one run the metric started at 0.48, peaked at 0.63 and training stopped at round 24, because with first_metric_only=False the default log loss also counts as a stopping metric, and it stopped improving first. Recall at the top 10% is a noisy metric on a small validation set (it moves in steps), so early stopping watches a jumpy curve; give it generous patience, or use a smoother metric such as PR-AUC for stopping and report recall at k only at the end. To watch only your own metric, set metric="None" in the model parameters so that the built-in metric is not computed.

Which metric should stop training?

  • If the decision depends on ranking, stop on aucpr or auc (built in for both libraries).
  • If you need probabilities, stop on logloss.
  • If you act at a threshold or with costs, use a custom cost metric as above, and tune the threshold afterwards (Thresholds).
  • For regression: rmse punishes large errors, mae is robust, mape and quantile losses fit relative or asymmetric errors.
flowchart LR
  O["Training objective<br/>smooth, differentiable"] --> M["Boosting rounds"]
  M --> V["Validation metric<br/>what you care about"]
  V -->|"stops improving"| S["Early stopping<br/>best_iteration"]

The objective drives learning; the evaluation metric decides when to stop.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a LightGBM custom eval metric function return?

    LightGBM expects (eval_name, eval_result, is_higher_better).
  2. 2. With xgb.train and a custom metric where a larger value is better, what must you do?

    XGBoost needs the direction stated with the maximize argument; otherwise early stopping works in the wrong direction.
  3. 3. Why might early stopping on recall at the top 10 percent behave erratically on a small validation set?

    With few positives, the metric moves in coarse steps and random swings, so use a larger patience or a smoother stopping metric.
  4. 4. What is the difference between the objective and the evaluation metric?

    Training needs gradients, evaluation does not.

Summary

  • Evaluation metrics decide early stopping and model choice; they can be custom even if the objective is standard.
  • XGBoost's native custom_metric returns a name and a value and needs maximize; LightGBM callables return a name, value and direction flag.
  • Pick the stopping metric to match the decision: ranking, probability, or cost at a threshold.
  • Use a noisy metric with patience, and report final results on a separate test set.

Next: Objectives for Counts, Quantiles and Ranking