Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 40 ยท Projects and Practice

Practice Exercises and Cheat Sheet

Test yourself with hands-on boosting exercises and worked solutions, then keep a one-page cheat sheet of parameters, workflow rules and the questions interviewers ask most often.

Reading is not enough: boosting skills come from running experiments and seeing what happens. This final lesson gives you practice problems of increasing difficulty, three of them with runnable solutions, followed by a cheat sheet to keep next to your notebook and the questions that come up most in interviews.

Exercises

Try each exercise yourself before opening the solution. Every solution is a complete program you can run and change.

Exercise 1: learning rate and tree count

Train an XGBClassifier on the data below with learning rates 0.3, 0.1 and 0.03, using early stopping on validation log loss. For each, report the best number of rounds and the validation log loss. Which learning rate would you choose, and why?

Python Exercise 1: solution

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In one run the best iterations were 37, 207 and 236 with validation log losses of 0.4281, 0.4281 and 0.4319. The usual answer: the validation losses are close, the smaller rates need many more trees, and the smallest rate has not gained anything here (it may even need more patience) while costing training time. Choose around 0.05 to 0.1 while exploring, and lower it only for the final model (Learning Rate and the Number of Trees).

Exercise 2: find the leak

The table below has a hidden leaky column. Train a model, rank the features by total gain, and identify the leak without looking at how the data was built. Then remove it and report how the validation AUC changes.

Python Exercise 2: solution

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In one run the suspect topped the gain ranking and the AUC fell from 0.93 to 0.65 without it. A retention call is made because a customer is about to leave (or after they left), so it is not available when the forecast is needed. Ask for every feature: "would this be known at prediction time?" (Pitfalls).

Exercise 3: a cost metric

A missed churner costs 50 and a retention offer sent to a loyal customer costs 4. Write a function that computes the total cost of a model at a threshold, find the best threshold on out-of-fold predictions, and compare it with the default 0.5 on a test set. (In one run the threshold was 0.08 and the test cost fell from 4,408 at 0.5 to 2,182, against 9,900 for flagging nobody.)

Python Exercise 3: solution

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

More exercises (no solution given)

  1. Tuning budget. Compare random search with Optuna on the same 30-trial budget and report the best cross-validated AUC of each (Bayesian Search with Optuna). Repeat with three different seeds. Is the difference larger than the seed-to-seed spread?
  2. Monotone constraints. Create data where price should decrease demand. Show that an unconstrained model sometimes violates this, then fix it (Dependence, Interactions and Monotone Constraints).
  3. Custom loss. Implement the pseudo-Huber loss as a custom objective for a target with a few huge outliers, verify the gradient numerically, and compare MAE against squared error (Custom Objective Functions).
  4. Extrapolation. Give a booster a target that grows with a feature beyond the training range. Fit a linear model first and let the booster learn the residuals. Does the combination extrapolate?
  5. Importance honesty. Add ten random noise columns to a dataset and compare which real features rank above the best noise column under weight, total_gain and permutation importance.
  6. Pipeline. Wrap data preparation and a booster in a scikit-learn Pipeline, save the booster in JSON, reload it and prove that predictions match (Saving, Loading and Versioning Models).
  7. Project. Take one of the three projects of this chapter and improve the headline metric. Document what helped, what did not, and the noise level of your measurements.

Cheat sheet

Workflow in ten lines

  1. Define the target and the decision it supports; choose the metric that reflects it.
  2. Split correctly: by time or by group when rows are related; stratify for rare classes.
  3. Build baselines (mean, majority class, linear model, last-period value).
  4. Train a default booster; check train versus validation scores.
  5. Fix the learning rate at 0.05 to 0.1; use early stopping with a large n_estimators.
  6. Tune tree size (max_depth, num_leaves) and evidence per leaf (min_child_weight, min_child_samples).
  7. Add subsampling and regularization (subsample, colsample_bytree, reg_lambda).
  8. Lower the learning rate for the final fit; average several seeds if stability matters.
  9. Choose thresholds, calibrate, and explain (permutation importance, SHAP).
  10. Save the model with its feature list, versions and threshold; monitor in production.

Parameters at a glance

Goal XGBoost LightGBM
Slower, smoother learning lower learning_rate, more trees lower learning_rate, more trees
Smaller trees lower max_depth, raise min_child_weight lower num_leaves, raise min_child_samples
Stronger regularization raise reg_lambda, gamma, reg_alpha raise reg_lambda, min_split_gain, reg_alpha
More randomness subsample < 1, colsample_bytree < 1 same, plus subsample_freq=1
Rare positives scale_pos_weight, aucpr scale_pos_weight or is_unbalance, average_precision
Categorical data enable_categorical=True with category dtype category dtype or categorical_feature
Speed tree_method="hist", device="cuda" max_bin, fewer leaves, num_threads
Reproducibility random_state, fixed n_jobs random_state, deterministic=True

Symptoms and first actions

Symptom Likely cause First action
Train score far above validation overfitting smaller trees, more regularization, lower learning rate
Both scores poor underfitting or weak features deeper trees, more rounds, better features
Validation too good to be true leakage inspect the top features, check time and group splits
Predictions flat beyond the data range trees cannot extrapolate model a ratio or difference, add a linear component
Great PR-AUC, poor alerts at 0.5 threshold not tuned choose a threshold on validation costs
Probabilities look inflated class weighting calibrate or drop the weights
Results change between runs unseeded randomness or threads set seeds, record versions

Questions interviewers like

  • Why does gradient boosting use the hessian as well as the gradient? The second derivative gives a Newton-style step and the leaf value -G/(H + lambda); it also tells the algorithm how much evidence a leaf holds.
  • How do XGBoost and LightGBM differ? Level-wise versus leaf-wise growth, different defaults and categorical handling, and LightGBM's gradient-based sampling and feature bundling; accuracy is usually close.
  • Why is boosting more prone to overfitting than a random forest? Each tree fits residuals of the previous ones, so the ensemble can keep reducing training error; early stopping and regularization are essential.
  • How do you handle imbalance? Right metric (PR-AUC), maybe weights, a tuned threshold, calibration if probabilities matter.
  • What are the weaknesses of built-in importance? Bias toward high-cardinality features, arbitrary credit sharing among correlated features, no direction, computed on training data.
  • Can trees extrapolate? No; leaves are constants. Detrend, model changes, or combine with a linear model.

Where to go next

You now have the whole path: from the algorithm (Gradient Boosting from Scratch in NumPy) through APIs, tuning, explanation, imbalance, custom losses and deployment. Go back to the track overview whenever you want to revisit a topic, and keep practicing on your own datasets: the habits of this track (honest splits, baselines, noise-aware comparisons) matter more than any single parameter.

Check your understanding

Quick check ยท 3 questions

of 3 answered You scored / 3 ยท perfect!

  1. 1. A feature dominates total gain and the validation AUC is 0.99 on a hard problem. What is your first move?

    A dominating feature with an unrealistically high score is the classic sign of leakage.
  2. 2. Which pair of parameters make trees smaller in XGBoost and LightGBM respectively?

    XGBoost is controlled by depth, LightGBM by the number of leaves.
  3. 3. What is the best way to compare two tuned models whose scores differ by 0.001?

    Without a noise estimate you cannot tell a real gain from luck.

Summary

  • Practice by running experiments: learning-rate sweeps, leak hunts, cost-based thresholds, constraints, custom losses.
  • Keep the ten-step workflow, the parameter table and the symptom table at hand.
  • The habits that matter most are honest splits, baselines, matching the metric to the decision, and measuring noise.

Next: XGBoost and LightGBM overview