Skip to content
Tutorials Center
My learning

Topics

Lesson 108 of 111 ยท Practice and Review

Exercises: Ensembles, Pipelines and Unsupervised

Ten advanced scikit-learn exercises with full solutions: random forests, boosting, stacking, pipelines, leakage, grid search, k-means clustering and PCA.

These ten exercises assume you are comfortable with the basics (see Exercises: Workflow, Regression and Classification) and move to the topics that separate a quick experiment from a trustworthy project: ensembles, pipelines that cannot leak, systematic tuning, and unsupervised methods. They come in three groups, each with an editor that holds the data and the split, so you can start straight away.

Three groups, ten problems

Ensembles Forests, boosting, stacking.
Pipelines Mixed data, leakage, tuning.
Unsupervised Clustering and PCA.

Each group has its own workspace with the data ready.

1 / 2

Within a group, solutions build on each other, so run them in order if you paste them under your own code. The exact scores may differ slightly on your machine, but the conclusions should not. Keep Random Forests, Pipelines and Clustering Basics open for reference.

Group 1: ensembles

The editor creates a binary classification problem with ten numeric features and splits it. Use X_train and y_train for every decision, and touch X_test only when an exercise says so.

Python Group 1 workspace: ensembles

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

1. Forest against boosting (Intermediate)

Compare RandomForestClassifier(n_estimators=100, random_state=0) and HistGradientBoostingClassifier(random_state=0) with 5-fold stratified cross-validation on the training data, scored by ROC AUC. Print the mean and standard deviation for each.

Show solution
Python
from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
for name, model in [("forest", RandomForestClassifier(n_estimators=100, random_state=0)),
                    ("hist boosting", HistGradientBoostingClassifier(random_state=0))]:
    scores = cross_val_score(model, X_train, y_train, cv=cv, scoring="roc_auc")
    print(f"{name:14s} mean {scores.mean():.3f}  std {scores.std():.3f}")

Output:

TEXT
forest         mean 0.931  std 0.015
hist boosting  mean 0.936  std 0.009

Both families are close on this synthetic problem. When two models are this near each other, the spread between folds matters more than the gap in the means; pick on other grounds such as speed, native handling of missing values, or ease of tuning (Histogram Gradient Boosting).

2. Which features matter? (Intermediate)

Fit the boosting model on the training data and compute permutation importance on the test set with n_repeats=5 and random_state=0. Print the three features with the largest mean importance. Why is permutation importance preferred over impurity-based importance for reporting?

Show solution
Python
from sklearn.inspection import permutation_importance

hgb = HistGradientBoostingClassifier(random_state=0).fit(X_train, y_train)
result = permutation_importance(hgb, X_test, y_test, n_repeats=5, random_state=0, scoring="roc_auc")
ranking = pd.Series(result.importances_mean, index=X.columns).sort_values(ascending=False)
print(ranking.head(3).round(3))

Output:

TEXT
f8    0.139
f0    0.116
f6    0.069
dtype: float64

Permutation importance measures how much the held-out score drops when one feature is shuffled, so it reflects what the model really relies on for unseen data. Impurity-based importance is computed on training data only and favors features with many distinct values. Remember that correlated features can share importance, so a low value does not always mean a useless feature (Feature Importance and Interpretation).

3. A free validation score (Intermediate)

Fit a RandomForestClassifier(n_estimators=200, oob_score=True, random_state=0) on the training data. Print the out-of-bag score and the test accuracy. Explain why the out-of-bag score is a fair estimate even though no validation set was set aside.

Show solution
Python
forest = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=0).fit(X_train, y_train)
print("out-of-bag accuracy:", round(forest.oob_score_, 3))
print("test accuracy      :", round(forest.score(X_test, y_test), 3))

Output:

TEXT
out-of-bag accuracy: 0.873
test accuracy      : 0.876

Each tree is trained on a bootstrap sample that leaves out roughly a third of the rows. For every training row, the forest collects votes only from trees that never saw it, which behaves like a held-out prediction. The two numbers agree closely, which is the typical picture.

4. How many boosting rounds? (Advanced)

Split X_train again into 80% fitting and 20% validation (stratified, random_state=1). Fit GradientBoostingClassifier(n_estimators=300, learning_rate=0.1, max_depth=2, random_state=0) on the fitting part. Use staged_predict_proba to compute the validation log loss after every round, print the best round and its loss, and print the loss after all 300 rounds. What does the difference tell you?

Show solution
Python
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import log_loss

X_fit, X_val, y_fit, y_val = train_test_split(X_train, y_train, test_size=0.2, stratify=y_train, random_state=1)
gbm = GradientBoostingClassifier(n_estimators=300, learning_rate=0.1, max_depth=2, random_state=0).fit(X_fit, y_fit)
losses = [log_loss(y_val, p) for p in gbm.staged_predict_proba(X_val)]
best_round = int(np.argmin(losses)) + 1
print("best round:", best_round, "| validation log loss:", round(min(losses), 4))
print("after 300 :", round(losses[-1], 4))

Output:

TEXT
best round: 214 | validation log loss: 0.3714
after 300 : 0.3761

The validation loss reaches its minimum and then rises, even though the training loss keeps falling: the later trees fit noise. The gap between the best and final loss is the price of not stopping early. In practice you let the library do this with n_iter_no_change or early_stopping, rather than choosing the round by hand (Gradient Boosting).

5. Does stacking help? (Advanced)

Build a StackingClassifier from a scaled logistic regression, a random forest and a histogram boosting model, with a logistic regression as the final estimator and cv=5. Fit it on the training data and print the test ROC AUC for the stack and for each base model fitted alone. Is the stack worth its extra cost here?

Show solution
Python
from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score

bases = [
    ("lr", make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))),
    ("rf", RandomForestClassifier(n_estimators=100, random_state=0)),
    ("hgb", HistGradientBoostingClassifier(random_state=0)),
]
stack = StackingClassifier(estimators=bases, final_estimator=LogisticRegression(), cv=5)
stack.fit(X_train, y_train)
print(f"{'stack':6s} {roc_auc_score(y_test, stack.predict_proba(X_test)[:, 1]):.3f}")
for name, model in bases:
    model.fit(X_train, y_train)
    print(f"{name:6s} {roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]):.3f}")

Output:

TEXT
stack  0.949
lr     0.849
rf     0.946
hgb    0.943

The stack is at best marginally better than its strongest member, and a gain this small is within the noise of a 450-row test set. Stacking trains each base model six times (once per fold plus once on all the data) and complicates deployment, so it pays off only when the base models make different kinds of errors and the gain is large enough to matter (Stacking and Blending Models).

Group 2: pipelines, leakage and tuning

The editor builds a small customer table with numeric and categorical columns, some missing values, and a target churned. The categorical columns are plan and city.

Python Group 2 workspace: churn table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

6. A complete tabular pipeline (Intermediate)

Build a ColumnTransformer named prep that imputes the numeric columns with the median and scales them, and one-hot encodes the categorical columns with handle_unknown="ignore". Put it in a pipeline with a LogisticRegression step named clf. Evaluate with 5-fold stratified cross-validation on all of X and y, scored by ROC AUC.

Show solution
Python
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold

prep = ColumnTransformer([
    ("num", Pipeline([("imputer", SimpleImputer(strategy="median")), ("scaler", StandardScaler())]), numeric),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
])
model = Pipeline([("prep", prep), ("clf", LogisticRegression(max_iter=1000))])
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(model, X, y, cv=cv, scoring="roc_auc")
print("ROC AUC per fold:", scores.round(3))
print(f"mean {scores.mean():.3f}  std {scores.std():.3f}")

Output:

TEXT
ROC AUC per fold: [0.754 0.759 0.729 0.759 0.716]
mean 0.743  std 0.018

Every learned quantity (the medians, the scaling, the category lists, the coefficients) is fitted inside each fold on that fold's training part, so the score is honest. The pipeline also accepts raw missing values and unseen categories, which is what you need in production (ColumnTransformer and Mixed Data).

7. Catch the leak (Advanced)

Create pure noise: 100 rows, 2,000 random features and random 0/1 labels (rng = np.random.default_rng(0)). There is nothing to learn, so a fair AUC is about 0.5. Do two experiments. In the first, select the 20 best features with SelectKBest(f_classif, k=20) on all the data and then cross-validate a logistic regression on the selected columns. In the second, put the selector and the classifier in one pipeline and cross-validate on the full data. Compare the scores and explain them.

Show solution
Python
from sklearn.feature_selection import SelectKBest, f_classif

rng = np.random.default_rng(0)
X_noise = rng.normal(size=(100, 2000))
y_noise = rng.integers(0, 2, 100)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

selected = SelectKBest(f_classif, k=20).fit_transform(X_noise, y_noise)
leaky = cross_val_score(LogisticRegression(max_iter=1000), selected, y_noise, cv=cv, scoring="roc_auc")
honest_model = Pipeline([("select", SelectKBest(f_classif, k=20)), ("clf", LogisticRegression(max_iter=1000))])
honest = cross_val_score(honest_model, X_noise, y_noise, cv=cv, scoring="roc_auc")
print("selection before CV :", round(leaky.mean(), 3))
print("selection inside CV :", round(honest.mean(), 3))

Output:

TEXT
selection before CV : 0.881
selection inside CV : 0.54

Selecting features on all the data lets the labels of the validation rows influence which columns survive. With 2,000 candidate columns, some correlate with the random labels by chance, and the model appears to find signal in pure noise. Inside the pipeline, the selection is redone on each training fold, and the score falls back to about chance. This is one of the most common ways to fool yourself (Nested Cross-Validation and Data Leakage).

8. Tune the whole pipeline (Advanced)

Using the model pipeline from exercise 6, run a GridSearchCV with 5-fold stratified cross-validation, scoring by ROC AUC, over the regularization strength clf__C in [0.01, 0.1, 1, 10] and the imputation strategy prep__num__imputer__strategy in ["mean", "median"]. Print the best parameters and the best cross-validated score. How many models were fitted in total?

Show solution
Python
from sklearn.model_selection import GridSearchCV

grid = {
    "clf__C": [0.01, 0.1, 1, 10],
    "prep__num__imputer__strategy": ["mean", "median"],
}
search = GridSearchCV(model, grid, cv=cv, scoring="roc_auc").fit(X, y)
print("best params:", search.best_params_)
print("best score :", round(search.best_score_, 3))
print("models fitted:", len(grid["clf__C"]) * len(grid["prep__num__imputer__strategy"]) * 5, "(+1 refit)")

Output:

TEXT
best params: {'clf__C': 0.1, 'prep__num__imputer__strategy': 'median'}
best score : 0.743
models fitted: 40 (+1 refit)

The double underscore walks down the nested names: step prep, then the num branch, then the imputer. The grid has 8 combinations, each fitted on 5 folds, so 40 fits plus a final refit on all the data. Look at search.cv_results_ to see whether the settings differ meaningfully before reading much into the winner. Since the same data was used to tune and to score, the best score is slightly optimistic; nested cross-validation would remove that bias (Hyperparameter Tuning).

Group 3: unsupervised learning

The editor generates 600 two-dimensional points from four Gaussian blobs. We keep the true blob numbers in y_true only so you can grade your clustering; a real project has none.

Python Group 3 workspace: blobs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

9. How many clusters? (Intermediate)

Run KMeans(n_clusters=k, n_init=10, random_state=0) for k from 2 to 7 and print the silhouette score for each. Pick the k with the highest silhouette and compute the adjusted Rand index against y_true. Does the silhouette find the true number of groups (4)?

Show solution
Python
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score, adjusted_rand_score

scores = {}
for k in range(2, 8):
    labels = KMeans(n_clusters=k, n_init=10, random_state=0).fit_predict(X_blobs)
    scores[k] = silhouette_score(X_blobs, labels)
    print(k, round(scores[k], 3))
best_k = max(scores, key=scores.get)
labels = KMeans(n_clusters=best_k, n_init=10, random_state=0).fit_predict(X_blobs)
print("best k:", best_k, "| adjusted Rand index:", round(adjusted_rand_score(y_true, labels), 3))

Output:

TEXT
2 0.53
3 0.626
4 0.567
5 0.53
6 0.442
7 0.356
best k: 3 | adjusted Rand index: 0.701

No, it does not. The silhouette is highest at k = 3 (0.626) and lower at the true k = 4 (0.567), because two of the four blobs overlap and look like one group to the metric. The adjusted Rand index of about 0.70 confirms the mismatch. Try a smaller cluster_std in the workspace and see whether the silhouette changes its mind. This is the normal state of affairs in unsupervised learning: the metric measures separation, not the truth, so combine it with other evidence and with domain knowledge (Evaluating Clusters).

10. How many components? (Advanced)

Load the 8 by 8 digits data (load_digits(return_X_y=True)) and fit a PCA on it. Print how many components are needed to keep 90% of the variance. Then reduce to 10 components, reconstruct the images with inverse_transform, and print the mean squared reconstruction error per pixel. What does the number of components trade off?

Show solution
Python
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA

X_digits, _ = load_digits(return_X_y=True)
pca = PCA().fit(X_digits)
cumulative = np.cumsum(pca.explained_variance_ratio_)
print("components for 90% variance:", int(np.argmax(cumulative >= 0.90)) + 1)

pca10 = PCA(n_components=10).fit(X_digits)
restored = pca10.inverse_transform(pca10.transform(X_digits))
print("variance kept by 10 components:", round(pca10.explained_variance_ratio_.sum(), 3))
print("reconstruction MSE per pixel  :", round(((X_digits - restored) ** 2).mean(), 3))

Output:

TEXT
components for 90% variance: 21
variance kept by 10 components: 0.738
reconstruction MSE per pixel  : 4.914

Fewer components give a smaller, faster, less noisy representation, but the reconstruction loses detail. The cumulative explained variance tells you exactly how much information each additional component buys, so you can stop where the gain flattens. Here we skipped scaling because all pixels share one unit; with mixed units you would standardize first (Principal Component Analysis).

Where to go next

Review any topic that felt uncertain: Gradient Boosting, Feature Selection, k-Means Clustering. When you are ready for a bigger task, try the full projects, such as Project: Credit Risk Scoring.

Summary

  • Compare ensembles by cross-validation and read the standard deviation next to the mean; the out-of-bag score is a free validation estimate for forests.
  • Permutation importance on held-out data tells you what a model uses; stop boosting where the validation loss is lowest.
  • Stacking adds cost and complexity, and it must earn its place with a clear gain.
  • Put imputing, scaling, encoding and selection inside one pipeline so they are fitted per fold; selection outside the fold manufactures signal from noise.
  • Tune the entire pipeline with nested parameter names, and remember the best score is slightly optimistic.
  • Choose k and the number of PCA components from several kinds of evidence, never from one number alone.

Next, prepare for conversations about all of this in Machine Learning Interview Questions.