Skip to content
Tutorials Center
My learning

Topics

XGBoost and LightGBM Tutorial

Learn XGBoost and LightGBM step by step: 40 lessons in 9 chapters with examples, quizzes, diagrams and projects.

Start learning →

of lessons completed

Getting Started

  1. 1 Why Gradient Boosting Wins on Tabular Data Meet gradient boosting, the idea behind XGBoost, LightGBM and CatBoost, see why it dominates spreadsheet-style data, and train your first two boosted models in the browser.
  2. 2 Installing and Setting Up XGBoost and LightGBM Install XGBoost, LightGBM and friends with pip or conda, check versions and CPU threads, set up reproducible environments and run a smoke test of both libraries.
  3. 3 Tabular Data for Boosting Learn what boosted trees need from a table: numeric and categorical columns, missing values, scaling that does not matter, and the data shapes both libraries accept.
  4. 4 Splits, Baselines and Metrics Evaluate XGBoost and LightGBM honestly: train, validation and test splits, simple baselines to beat, the right metric for the task, time-based splits and how to spot leakage.
  5. 5 First Models with XGBoost and LightGBM Train XGBoost and LightGBM side by side for classification and regression, compare the key parameters, read predictions and probabilities, and see how fit time and accuracy compare.

Gradient Boosting Basics

  1. 6 Decision Trees and Ensembles Understand the building block behind XGBoost and LightGBM: how a decision tree splits data, why one tree over- or underfits, and how bagging and boosting combine many trees.
  2. 7 Gradient Boosting from Scratch in NumPy Build a gradient boosting regressor yourself with small decision trees and residuals, watch the error fall round by round, and compare it with the real XGBoost library.
  3. 8 Loss, Gradients and Hessians See how boosting libraries turn any loss function into per-row gradients and Hessians, how leaf values come from them, and write a custom objective that matches the built-in one.
  4. 9 Regularization and Subsampling Learn the split gain formula, how lambda, alpha and gamma prune trees, and how row and column subsampling add randomness that fights overfitting in XGBoost and LightGBM.
  5. 10 Histogram Splits and Tree Growth Strategies Understand why modern boosters bin features into histograms, how level-wise (XGBoost) and leaf-wise (LightGBM) growth differ, and what max_bin, max_depth and num_leaves control.

XGBoost API

  1. 11 XGBoost Scikit-learn API Use XGBClassifier and XGBRegressor like any scikit-learn estimator: fit with an eval set, read learning curves, use pipelines and cross-validation, and clone, tune and inspect models.
  2. 12 DMatrix and the Native train API Learn XGBoost's native interface: build a DMatrix with labels, weights and feature names, pass a params dict to xgb.train, watch evals, predict, inspect trees and set a base margin.
  3. 13 XGBoost Categorical and Missing Values Handle missing values with XGBoost's learned default direction and use enable_categorical for native category splits, compared against one-hot and ordinal codes, with real runnable examples.
  4. 14 XGBoost Objectives and Evaluation Metrics Pick the right XGBoost objective for regression, counts, binary, multiclass and ranking, understand how it differs from eval_metric, and add a custom metric to xgb.train.
  5. 15 Early Stopping, Cross-Validation and Saving Models Pick the number of boosting rounds with xgb.cv and early stopping, then save and reload XGBoost models in JSON or UBJ and with the scikit-learn API, and avoid the pickle pitfalls.

LightGBM and CatBoost

  1. 16 LightGBM Scikit-learn API Train LightGBM with LGBMClassifier and LGBMRegressor, use callbacks for early stopping and logging, read evals_result_, and set the parameters that matter most for speed and overfitting.
  2. 17 LightGBM Dataset and Categorical Features Use lgb.Dataset and lgb.train for the native LightGBM API, run lgb.cv, and handle categorical columns and missing values directly, with the parameters that control category splits.
  3. 18 CatBoost and Ordered Boosting Learn what makes CatBoost different from XGBoost and LightGBM: ordered target statistics for categories, ordered boosting, symmetric trees and its API, with runnable stand-ins for the ideas.
  4. 19 Choosing a Booster Compare XGBoost, LightGBM, CatBoost and scikit-learn's HistGradientBoosting on speed, accuracy, categorical handling, GPU support and defaults, and learn a practical way to decide.

Hyperparameter Tuning

  1. 20 A Sensible Tuning Workflow Learn how to tune gradient boosting models without fooling yourself: baselines, train/validation/test splits, early stopping, search spaces and how much tuning is worth.
  2. 21 Learning Rate and the Number of Trees Understand the trade-off between learning rate and number of boosting rounds, read learning curves, and use early stopping in XGBoost and LightGBM to pick the tree count.
  3. 22 Tree Complexity Parameters Learn which parameters limit tree size in XGBoost and LightGBM: max_depth, min_child_weight, gamma, num_leaves and min_child_samples, and how to tune them in the right order.
  4. 23 Bayesian Search with Optuna Go beyond grid and random search: define an Optuna study, sample hyperparameters with TPE, prune bad trials and read the results for XGBoost and LightGBM models.
  5. 24 Tuning LightGBM and XGBoost Side by Side A parameter translation table between XGBoost and LightGBM, native cv with early stopping in both libraries, and starting ranges that give a strong first model.

Feature Importance and SHAP

  1. 25 Feature Importance: Gain, Weight and Cover Learn the built-in importance types of XGBoost and LightGBM (weight, gain, cover, split), why they disagree, and how to plot and interpret them without being misled.
  2. 26 Permutation Importance Measure how much a trained model really relies on each feature by shuffling it on held-out data. Includes repeats, correlated features, grouped permutation and limitations.
  3. 27 SHAP Values for Tree Models Understand SHAP values as fair per-prediction contributions, verify their additivity with pred_contribs, and use shap.TreeExplainer for local and global explanations.
  4. 28 Dependence, Interactions and Monotone Constraints See how a boosted model reacts to one feature with partial dependence, find interactions with pred_interactions, and force sensible monotone effects in XGBoost and LightGBM.

Imbalanced Data and Custom Losses

  1. 29 Class Imbalance and Sample Weights Handle rare positives in XGBoost and LightGBM with scale_pos_weight, sample weights and is_unbalance, and learn what weighting does to recall, precision and probability calibration.
  2. 30 Thresholds and Metrics for Imbalanced Data Pick the decision threshold of a boosted classifier with precision-recall curves, F-scores and costs, using out-of-fold predictions so the choice is honest and stable.
  3. 31 Custom Objective Functions Write your own loss for XGBoost and LightGBM by returning the gradient and hessian: verify squared error, build an asymmetric loss and a focal loss, and avoid the common traps.
  4. 32 Custom Evaluation Metrics Write a custom evaluation metric for XGBoost and LightGBM, make early stopping watch the number you care about, and learn the differences in signatures and the higher-is-better flag.
  5. 33 Objectives for Counts, Quantiles and Ranking Choose the right built-in objective: Poisson and Tweedie for counts and claims, quantile and absolute error for intervals and outliers, and rank objectives for sorted result lists.

Deployment and Pitfalls

  1. 34 Saving, Loading and Versioning Models Save boosted models in portable formats (JSON, UBJ, text), compare them with pickle and joblib, verify round trips, and version a model together with its feature list and metadata.
  2. 35 Fast and Scalable Inference Make boosted-tree predictions fast and predictable: batch versus single-row latency, limiting trees, threads, model size, exporting to other runtimes and a pure Python scorer.
  3. 36 Pitfalls: Leakage, Extrapolation and Reproducibility Avoid the most expensive boosting mistakes: target leakage and time-split errors, trees that cannot extrapolate, drift between training and serving, and results that change between runs.

Projects and Practice

  1. 37 Project: Fraud Detection Build a complete fraud model on generated transactions: time-based split, imbalance, PR-AUC, a cost-based threshold, top-k review lists and per-alert explanations with SHAP contributions.
  2. 38 Project: House Price Regression Predict home prices with LightGBM on a generated dataset: baselines, categorical features, a log target, early stopping, error analysis by price band and feature contributions.
  3. 39 Project: Sales Forecasting with Lag Features Turn a daily sales series into a supervised table with lags, rolling means and calendar features, validate with a time split, and fix the trend problem by predicting a ratio.
  4. 40 Practice Exercises and Cheat Sheet Test yourself with hands-on boosting exercises and worked solutions, then keep a one-page cheat sheet of parameters, workflow rules and the questions interviewers ask most often.