Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 40 ยท Deployment and Pitfalls

Fast and Scalable Inference

Make boosted-tree predictions fast and predictable: batch versus single-row latency, limiting trees, threads, model size, exporting to other runtimes and a pure Python scorer.

Training is a one-time cost. Prediction runs millions of times, often inside a request with a strict latency budget. Boosted trees are fast, but "fast" depends on how you call them. This lesson explains where the time goes, how to batch, how to shrink the model, and what your options are when the serving environment cannot install XGBoost or LightGBM at all.

Where prediction time goes

A prediction walks every tree from root to leaf and sums the leaf values. The cost is roughly rows x trees x depth. In practice you pay three more costs:

  • Fixed overhead per call. Converting a DataFrame, validating shapes and crossing the Python-to-C boundary costs tens or hundreds of microseconds, regardless of how many rows you send.
  • Data conversion. Passing a DataFrame with mixed types is slower than passing a float32 NumPy array.
  • Threading. Multi-threaded prediction pays off for large batches but can slow a single-row call.

The first fact has a large consequence: batch your rows. One call with 1,000 rows is far cheaper than 1,000 calls with one row.

Python Row by row versus one batch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In one run in the browser, 500 single calls took about 3.7 seconds and one batch call about 55 milliseconds, roughly 67 times faster. The absolute times depend on your machine, but batching is usually one to two orders of magnitude faster. For a web service this means collecting concurrent requests for a few milliseconds and scoring them together (micro-batching), or scoring offline in bulk and storing the results.

Trade accuracy for speed: fewer trees

Latency grows linearly with the number of trees. You can predict with only the first k trees by passing iteration_range=(0, k) (XGBoost) or num_iteration=k (LightGBM). Often the last hundred trees add almost nothing:

Python How many trees do we really need?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In one run the validation AUC was 0.914 with 25 trees, 0.927 with 50, 0.940 with 100, 0.9445 with 200 and 0.9456 with 400: the curve flattens long before the last tree, while prediction time grows in proportion to the tree count (add a timer around the call to see it on your machine). A compromise is to train with a higher learning rate (fewer trees), or to pick the smallest number of trees within 0.001 AUC of the best. Also remember that early stopping already selected a good count (Learning Rate and the Number of Trees).

Other levers

  • Smaller trees. max_depth=4 is much cheaper than depth 8 at the same tree count.
  • Float32 inputs. Both libraries store features as 32-bit floats internally; give them float32 arrays to avoid a conversion.
  • Thread count. For large batches use all cores (n_jobs=-1 or nthread). For single-row latency set n_jobs=1, and run several model instances (processes) behind a load balancer rather than one multi-threaded one.
  • Inplace prediction. booster.inplace_predict(array) in XGBoost skips the DMatrix construction for dense arrays.
  • GPU. device="cuda" helps very large batches (millions of rows) but loses for small ones because of transfer costs.
  • Feature computation usually dominates. Profile the whole request. Building features from a database often costs more than the model.

Leaving Python: other runtimes

When the serving stack is Java, C++, mobile or a database, you can export the model:

  • ONNX (onnxmltools, skl2onnx, onnxruntime): run XGBoost and LightGBM in a single portable runtime, in many languages.
  • Treelite / Hummingbird / m2cgen / lleaves: compile trees into optimized C code or tensor operations. The compiled code can be several times faster than the library predictor.
  • Native bindings: XGBoost4J (JVM), the LightGBM C API and R and C# bindings let you load the same model file directly.

These tools need a build environment and cannot run in this site's browser sandbox, so treat them as pointers. Always repeat the round-trip test from Saving and Loading: predictions from the exported runtime must match the original within a small tolerance.

A pure Python scorer

The simplest "export" is no export at all: a tree ensemble is just a list of nested comparisons. You can read the trees from trees_to_dataframe() and evaluate them with plain Python, which also makes the algorithm concrete:

Python Scoring one row without calling the library

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The comparison is done in 32-bit floats because XGBoost stores features and thresholds that way; with 64-bit values a row very close to a threshold can take the other branch. This version ignores missing values and categorical splits, which a production scorer must handle (the Missing column of the DataFrame gives the default direction). It is a teaching tool, and a reminder that a deployed model is just data: thresholds and leaf values.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is one call with 1,000 rows faster than 1,000 calls with one row?

    The per-call overhead is paid once per call, so batching spreads it over many rows.
  2. 2. How can you use only the first 100 trees of a trained XGBoost model at prediction time?

    iteration_range selects the boosting rounds used for the prediction (LightGBM: num_iteration).
  3. 3. For single-row, low-latency serving, which thread setting is usually best?

    Thread coordination costs more than it saves for one row; scale out with processes instead.
  4. 4. Which is a valid way to run a trained model outside Python?

    Several export and binding options exist; always verify that predictions match the original.

Summary

  • Prediction cost is trees x depth per row plus a fixed per-call overhead; batch your rows.
  • Use fewer or shallower trees when accuracy has flattened; check the trade-off on validation data.
  • Use float32 arrays, n_jobs=1 for single rows, many threads for big batches, and measure the whole request, since feature building often dominates.
  • ONNX, Treelite and native bindings allow serving outside Python; always re-verify the predictions.

Next: Pitfalls: Leakage, Extrapolation and Reproducibility