Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 111 ยท Trees and Ensembles

Stacking and Blending Models

Combine different models with StackingClassifier and StackingRegressor: how a meta-learner works, why cross-fitted predictions matter, and when ensembles are worth the cost.

Random forests average many trees; voting classifiers average a few different models with equal say. Stacking goes one step further: it trains a second model, the meta-learner, whose job is to learn how much to trust each base model. Perhaps the linear model is reliable in the middle of the data and the forest is better at the edges, and the meta-learner can discover that from the data instead of you guessing weights.

Stacking is a favorite of competition winners because it squeezes out the last bit of accuracy. It is also easy to do wrongly, in a way that quietly inflates your score. This lesson shows the mechanics, the one trick that keeps it honest (cross-fitted predictions), and a realistic look at when it pays off.

Stacking in three steps

Base models Several different models, each trained on the data.
Their predictions Out-of-fold predictions become new features.
Meta-learner A simple model learns how to combine them.

The meta-learner never sees the original features unless you ask it to.

1 / 2

How a stack is built

Suppose you have three base models. For every training row, each model produces a prediction (a class probability or a number). Those three predictions are placed side by side as three new columns, and the meta-learner is trained to predict the true label from them.

flowchart LR
    X["training data"] --> A["model A"]
    X --> B["model B"]
    X --> C["model C"]
    A -->|"out-of-fold<br/>predictions"| M["meta-learner"]
    B -->|"out-of-fold<br/>predictions"| M
    C -->|"out-of-fold<br/>predictions"| M
    M --> P["final prediction"]

The meta-learner sees only what the base models say, one column per model.

scikit-learn does all of this in StackingClassifier and StackingRegressor. You give them a list of (name, estimator) pairs and a final_estimator. The meta-learner should usually be simple: logistic regression for classification, ridge regression for regression. A powerful meta-learner has little to learn from a few columns and mostly adds the risk of overfitting.

The first example stacks a scaled logistic regression, a random forest and a k-nearest-neighbors model, and compares the result with each of them using 5-fold cross-validation (Cross-Validation covers the scoring call).

Python A first stacking classifier

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the gap between the stack and the best single model. It is typically small, a fraction of a percentage point. That is the normal size of the prize, and later in the lesson you will see why.

Why the predictions must be cross-fitted

There is a trap hiding in the diagram. If you trained each base model on the full training set and then asked it to predict those same rows, the predictions would be far too good. A random forest, for example, has memorized its training rows and is nearly perfect on them. The meta-learner would see columns that look almost infallible, learn "trust the forest completely", and then fail on new data where the forest is merely good.

The fix is cross-fitted (out-of-fold) predictions: split the training data into folds, and for each row use a model that was trained on the other folds, so it never saw that row. These honest predictions have the same quality as predictions on new data, which is exactly what the meta-learner needs to learn from. This is what the cv=5 argument of StackingClassifier does.

Python In-sample vs cross-fitted predictions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The in-sample column makes the forest look perfect, so the meta-learner gives it an enormous weight. The cross-fitted column tells the truth about how reliable the forest is, and the weight shrinks accordingly. In this single-column case the test accuracy may not change much (the weight only rescales the same ranking), but with several base models the inflated weight on the most overfit model is exactly what hurts.

Never build a stack by hand from model.predict(X_train) of models trained on X_train. Use StackingClassifier/StackingRegressor, or cross_val_predict, so the meta-learner only sees out-of-fold predictions.

Stacking for regression, and reading the weights

StackingRegressor works the same way. A good use case is two models that fail in different ways: a linear model captures overall trends but cannot bend, a forest captures bends but cannot extrapolate smoothly. Using Ridge(positive=True) as the meta-learner gives non-negative weights you can read like a recipe: how much of each model goes into the final prediction.

Python Learned weights of a stacking regressor

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The data mixes straight trends (which the linear model captures and a small forest approximates poorly) with a sine wave (which the forest captures and the linear model cannot). The two models score almost the same, and the stack edges past both by a small margin, using roughly 0.6 of the linear model and 0.4 of the forest. That is a realistic gain: real, but small.

Two more options are worth knowing:

  • passthrough=True also hands the original features to the meta-learner, next to the predictions. That gives it more room to correct the base models, at the price of a bigger, easier-to-overfit second stage.
  • For classifiers, stack_method="auto" uses predict_proba when available. Probabilities carry more information than hard class labels, so leave it alone.

Blending: the simpler cousin

Blending is stacking with a hold-out set instead of cross-fitting. You split the training data in two: train the base models on part A, predict part B, and train the meta-learner on those predictions for part B. It is easy to code and impossible to leak, but it wastes data, because the base models never see part B and the meta-learner sees only part B. With cross-fitting every row serves both roles, so on small and medium data sets scikit-learn's stacking is the better default.

The name is used loosely online. Some people say "blending" for any weighted average of predictions. Both meanings share the principle: weights must be learned from predictions on rows the base models did not train on.

When ensembles help and when they just add cost

A stack is only as good as the diversity of its members. If two models make the same mistakes, combining them adds nothing. The test below makes this concrete. A logistic regression and a ridge classifier are both linear models, and they agree on almost every row, so stacking them barely moves the score.

Python Similar models add nothing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Before reaching for a stack, run through this checklist:

  1. Is there a diverse set of strong models? A linear model, a forest, a boosted model and a nearest-neighbors model disagree in useful ways. Five boosted models with different seeds do not.
  2. Is the gain bigger than the noise in your estimate? Cross-validation scores wobble by a few tenths of a percentage point; a gain of that size is not real.
  3. Can you afford it? A stack with 5-fold cross-fitting trains every base model six times (five folds plus the final fit), and predicting runs all of them. Latency, memory and maintenance all multiply.
  4. Can you explain it? Regulated or high-stakes settings often prefer one inspectable model over a slightly more accurate pile of models.

On many real tabular problems a single tuned histogram gradient boosting model is within a hair of the best stack. Treat stacking as a final polish after the data, features and a single good model are in order, not as a first move.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the meta-learner in a stacking model take as input?

    Each base model's prediction becomes one new column. The meta-learner learns how to combine those columns (unless passthrough=True adds the raw features too).
  2. 2. Why are the meta-learner's training inputs cross-fitted (out-of-fold) predictions?

    A model has seen its training rows, so its predictions there are optimistic. Out-of-fold predictions come from models that never saw the row, mimicking real new data.
  3. 3. Which pair of base models is most likely to give a useful stack?

    Stacking helps when models make different mistakes. A linear model and a tree ensemble have very different strengths.
  4. 4. How does blending differ from stacking in scikit-learn's sense?

    Blending is the hold-out version: simple and leak-free, but it wastes data compared with cross-fitting.
  5. 5. Which meta-learner is the usual sensible choice?

    The meta-learner sees only a handful of columns, so a simple model suffices and overfits less.

Summary

  • Stacking trains a meta-learner on the predictions of several base models; StackingClassifier and StackingRegressor automate it.
  • The meta-learner must be trained on out-of-fold predictions (cv=), otherwise overfit base models get too much weight.
  • Keep the meta-learner simple; Ridge(positive=True) gives readable weights for regression.
  • Blending uses a single hold-out set: simpler, but it wastes data.
  • Gains need diversity among the base models and are usually small; weigh them against training time, latency and complexity.

Next, look inside your models with Feature Importance and Interpretation.