Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 40 ยท Getting Started

Tabular Data for Boosting

Learn what boosted trees need from a table: numeric and categorical columns, missing values, scaling that does not matter, and the data shapes both libraries accept.

One reason boosted trees are so popular is how little preparation they ask for. You do not scale features, you rarely need to impute, and categories can often be passed straight in. But "little" is not "none". This lesson shows what the libraries accept, what they do on their own, and where you still have to be careful.

We will build a small customer table with a numeric column, a categorical column and some missing values, and feed it to both libraries.

The shapes of data

Both libraries accept:

  • NumPy arrays of shape (rows, columns);
  • pandas DataFrames, which keep column names (useful for plots and importances);
  • SciPy sparse matrices, ideal for one-hot or text-count features;
  • their own dataset classes: xgboost.DMatrix and lightgbm.Dataset, which pre-bin the data for speed (covered in later lessons).

Targets are a 1-D array: numbers for regression, class labels for classification. The scikit-learn wrappers also accept pandas Series.

Scaling does not matter

A tree asks questions like "is income below 41,500?". Multiplying the column by 1000 or adding 7 only moves the threshold, so the same rows end up on each side and the tree is identical. Boosting inherits that property. Prove it:

Python Scaling a column changes nothing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The difference is tiny (histogram bin edges are computed from quantiles, so the result is the same up to rounding). Any monotonic transformation (log, square root, rank) is also harmless for the features. That is very different from linear models and neural networks, which need scaling. Note that the target is different: transforming y (for example log1p) does change what the model learns, and is often useful for skewed targets.

Missing values are first-class

Both libraries treat NaN as information. During training, each split learns a default direction: rows with a missing value go left or right, whichever lowers the loss. So you can usually leave gaps alone instead of filling them with the mean. Filling is still sometimes useful, but it can also erase a signal ("the field was blank" may predict the outcome).

Python Missing values left as NaN

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The third probe row, with a missing income, receives a high probability. The models learned that "missing" is itself a clue. A mean fill would have hidden that.

Missing means missing to the model, so do not mix in other markers. Values such as -999, 0 for "unknown" or the string "N/A" are treated as real numbers or break the conversion. Convert them to NaN first.

Categorical columns

You have three options for text categories such as plan = "free" | "pro" | "team".

  1. Native categorical support. Convert the column to pandas category dtype. LightGBM uses it automatically; XGBoost needs enable_categorical=True. This is usually the best choice, and later lessons explain how the split search works.
  2. One-hot encoding with pd.get_dummies or scikit-learn's OneHotEncoder. Fine for a few categories, wasteful for hundreds.
  3. Ordinal or target encoding. Ordinal codes (0, 1, 2) can work for trees but invent an order; target encoding is powerful but leaks the label unless it is computed inside cross-validation.
Python Native categories in both libraries

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When predicting, always give the new data the same category list as training (pd.Categorical(..., categories=...)). Otherwise the integer codes can shift and the model reads the wrong categories. We return to this in XGBoost categorical and missing values.

Column types at a glance

What each column needs

Numbers Pass as is. No scaling.
Categories Use category dtype or one-hot.
Gaps Leave NaN. Convert odd markers.
Dates Split into year, month, weekday.

Text and images need their own features first.

1 / 2

Raw dates and free text are not usable as they are. Turn a date into numbers (year, month, day of week, days since an event) and text into counts or embeddings before boosting.

Things that still need care

  • Rare high-cardinality categories (user IDs, zip codes) invite overfitting; group rare values or use target encoding inside cross-validation.
  • Duplicated rows across train and test inflate scores.
  • Target leakage: a column computed after the outcome (for example "refund issued" when predicting churn) gives a model an unfair clue. Boosting will find and love it.
  • Very skewed targets often train better after log1p.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is feature scaling unnecessary for tree boosters?

    Only the order of values matters for a split, and rescaling keeps the order.
  2. 2. How should you mark a missing value?

    Both libraries learn a default direction for NaN. Placeholder numbers are treated as real values.
  3. 3. What must you keep consistent between training and prediction for categorical columns?

    Categories are stored as integer codes, so a different level list shifts the codes and misreads values.
  4. 4. Which transformation of the TARGET is often helpful for a right-skewed regression target?

    A log transform reduces the pull of huge values, and you invert it (expm1) after predicting.

Summary

  • Boosters take arrays, DataFrames and sparse matrices; no scaling is needed for features.
  • Leave missing values as NaN; the model learns where they belong.
  • Use category dtype (enable_categorical=True in XGBoost) or one-hot for categories, and keep levels consistent.
  • Convert dates and text to numbers first; watch for rare categories and leakage.

Next: Splits, Baselines and Metrics