Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 111 ยท Trees and Ensembles

AdaBoost and Boosting from Scratch

Understand AdaBoost by building it: reweight samples, train decision stumps, combine them with weights, then use scikit-learn's AdaBoostClassifier and AdaBoostRegressor.

Boosting turns many weak models into one strong model. A weak learner is a model that is only a little better than guessing, such as a decision tree with a single split (a stump). Boosting trains these learners one after another, and each new learner concentrates on the examples that the earlier ones got wrong. AdaBoost (adaptive boosting) was the first practical boosting algorithm, and it is still the cleanest way to see the idea in action.

It also has a natural place between random forests and gradient boosting. A random forest trains its trees independently and averages them. AdaBoost trains them in sequence, and every tree depends on the mistakes of the ones before it. Gradient boosting generalizes the same sequential idea from "reweight the samples" to "fit the remaining error".

How AdaBoost works

Start equal Every sample has the same weight.
Fit a weak learner A stump trained on weighted data.
Raise the misses Wrong samples get more weight.
Weighted vote Better stumps get a bigger say.

Repeat for a fixed number of rounds.

1 / 2

The algorithm in plain words

Take a binary problem with labels written as -1 and +1. Give each of the n training samples the same weight 1/n. Then repeat these steps for M rounds:

  1. Fit a weak learner to the training data, using the sample weights. The learner tries to minimize the weighted error, so heavy samples matter more.
  2. Compute the learner's weighted error err, the total weight of the samples it got wrong.
  3. Give the learner a vote strength alpha = 0.5 * ln((1 - err) / err). A learner with a small error gets a large alpha. A learner that is no better than a coin flip (err = 0.5) gets alpha = 0, and a learner worse than a coin flip gets a negative one.
  4. Update the sample weights: multiply each weight by exp(-alpha * y * prediction). Correct samples (where y * prediction = 1) shrink, wrong ones (where it is -1) grow. Then divide by the sum so the weights add up to 1 again.

The final prediction is the sign of the weighted vote: sign(alpha_1 * h_1(x) + alpha_2 * h_2(x) + ...). That is all there is to it. The weight update is the clever part: after step 4, the last learner's weighted error on the new weights is exactly 0.5, so the next learner is forced to find something the previous one could not do.

Build it from scratch

The first example implements those four steps with NumPy. The weak learner is a stump found by brute force: for every feature, every threshold and both orientations, keep the one with the lowest weighted error. The data is the two interleaved half-circles from make_moons, which no single split can separate.

Python AdaBoost with hand-written stumps

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Watch the printed numbers. The first stump already gets about 80% of the training data right, which gives a weighted error of 0.2 and a large vote (alpha about 0.69). Later stumps work on a reweighted problem, so their weighted errors creep towards 0.5 (0.364 in round 10, 0.428 in round 30) and their votes shrink. The largest sample weight also grows from under 1% to several percent: those are the hard points near the boundary that the ensemble keeps getting wrong. Accuracy rises as stumps are added, from 0.76 on the test set for one stump to 0.90 for thirty.

Each stump is a poor model, yet their weighted vote draws a staircase-shaped boundary that follows the curved classes. That is the bias-reducing power of boosting: a simple learner class becomes much more flexible once you stack many of them.

AdaBoost in scikit-learn

Writing the loop yourself is for understanding. In practice use AdaBoostClassifier and AdaBoostRegressor. They accept any weak learner that supports sample weights, and the default is a depth-1 tree.

Two parameters matter most. n_estimators is the number of rounds. learning_rate multiplies every learner's vote: a smaller value needs more rounds to reach the same fit, but often generalizes slightly better, the same trade-off you will meet in gradient boosting. Scikit-learn's multi-class algorithm is called SAMME, and it scales the votes as ln((1 - err) / err) + ln(K - 1) without the 0.5 from the formula above, which is why its estimator weights look about twice as large as ours for a two-class problem.

Python AdaBoostClassifier and AdaBoostRegressor

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The classifier shows the expected pattern: a single stump gets about 0.76, five stumps 0.87 and thirty 0.89, and it stops improving after that because the data contains noise. Lowering the learning rate to 0.1 with the same 30 rounds drops the score to 0.84, because the votes are too small to build the boundary yet. The regressor result is an honest reminder that boosting is not magic. A depth-3 tree is already a good model for a smooth sine wave with little noise, and AdaBoost on top of it gives no gain on this split. Boosting pays off when a single weak learner cannot capture the shape of the data, not on every dataset.

Strengths and weaknesses

  • It needs little tuning. Two parameters and a weak learner are enough to get a strong baseline on tabular data.
  • It is sensitive to noise and outliers. Mislabeled samples are always wrong, so their weight keeps growing and later learners spend their effort on them. If your labels are noisy, cap n_estimators, lower learning_rate, or prefer a bagging method such as random forests.
  • It trains sequentially. You cannot build the trees in parallel as in a forest.
  • It usually loses to modern gradient boosting. For serious tabular work, try histogram-based gradient boosting or XGBoost and LightGBM first. AdaBoost remains a useful teaching tool and a decent quick baseline.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. After a round of AdaBoost, which samples get a higher weight?

    The update multiplies each weight by exp(-alpha * y * prediction). Wrong predictions make the exponent positive, so those weights grow and the next learner has to pay attention to them.
  2. 2. A weak learner has a weighted error of exactly 0.5. What vote strength alpha does it receive?

    alpha = 0.5 * ln((1 - 0.5) / 0.5) = 0.5 * ln(1) = 0. A learner that is no better than guessing has no influence on the final vote.
  3. 3. Which description of boosting versus bagging is correct?

    Boosting is sequential and corrective, which mainly reduces bias. Bagging averages independent models trained on bootstrap samples, which mainly reduces variance.
  4. 4. Your labels contain many mistakes. Why can AdaBoost struggle?

    Because the weight update rewards focusing on misclassified samples, noisy labels attract more and more attention. Fewer rounds, a smaller learning rate or a bagging method are safer then.

Summary

  • Boosting combines many weak learners trained one after another; each round reweights the samples to focus on earlier mistakes.
  • AdaBoost gives each learner a vote alpha = 0.5 * ln((1 - err) / err) and predicts the sign of the weighted vote.
  • Sample weights are multiplied by exp(-alpha * y * prediction) and renormalized, so hard samples gain weight.
  • In scikit-learn, tune n_estimators, learning_rate and the weak learner's depth; stumps are the default.
  • AdaBoost is sensitive to label noise and is usually beaten by gradient boosting on tabular data.

Next: Gradient Boosting replaces sample reweighting with fitting the remaining errors directly.