Skip to content
Tutorials Center
My learning

Topics

Lesson 42 of 77 ยท Databases and Data Interview Questions

Machine Learning: Regularization, Validation, Leakage and Ensembles

Prepare machine learning interviews: L1 and L2 regularization, cross-validation, data leakage, gradient descent, bagging versus boosting, all run for real.

Machine learning interviews rarely ask you to derive an algorithm. They ask you to diagnose: "The model has 99% training accuracy and 70% on new data, what do you do?", "How did you validate it?", "Your cross-validation score is great but production is terrible, why?" Good answers name the cause (overfitting, leakage, a bad learning rate) and a fix, and are backed by something you have actually run.

The previous data lesson, Data and Python Questions, already covers the bias-variance picture and precision, recall and F1. This lesson goes one step further: regularization, honest validation and data leakage, gradient descent, and ensembles. Every program uses the site's real scikit-learn and NumPy in the Python runner (the first run downloads the packages, so allow a few seconds). For full tutorials, see Machine Learning and Data Science.

A diagnosis checklist

Overfitting Great on training, poor on new data.
Validation Test only on data the model never saw.
Leakage Information from the test set sneaks in.
Optimization Learning rate, steps, scaling.

Name the cause before you name the fix.

1 / 2

What is regularization, and how do L1 and L2 differ?

Short answer: Regularization adds a penalty on large weights to the loss function, so the model prefers simpler solutions and overfits less. L2 (Ridge) penalizes the sum of squared weights: weights shrink smoothly toward zero but rarely reach it. L1 (Lasso) penalizes the sum of absolute values: it can push weights exactly to zero, so it also performs feature selection. The strength is a hyperparameter (alpha in scikit-learn, or ฮป), chosen by cross-validation. Elastic Net combines both.

Why does it help? A flexible model can fit noise with huge, opposing coefficients. Penalizing size removes that freedom: a little more bias, much less variance. Remember that regularized models need scaled features, otherwise the penalty treats a feature measured in thousands differently from one measured in fractions.

The experiment below builds data with 10 features of which only 3 matter (positions 4, 7 and 9) and fits three models.

Python Regularization: Ridge versus Lasso

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Plain least squares assigns small non-zero weights to every useless feature. Ridge shrinks all weights (including the real ones, which is the cost of the penalty) but keeps every feature. Lasso sets the useless weights to exactly zero and keeps the three informative features, so it has found the right ones. That contrast, "Lasso selects, Ridge shrinks", is the sentence interviewers want.

How should you validate a model, and what is cross-validation?

Short answer: Never evaluate on the data you trained on. Split the data into train, validation (to tune hyperparameters) and test (touched once, at the end). With small data, use k-fold cross-validation: split into k parts, train on k-1, validate on the remaining one, rotate, and average the k scores. It uses all the data for both training and evaluation and gives you a spread as well as a mean. Use stratified folds for classification so each fold keeps the class proportions, and time-based splits (never shuffled) for time series, because you must not train on the future.

What is data leakage?

Short answer: Leakage is when information from outside the training set (the test labels, future data, or a feature that is a proxy for the answer) influences the model, so your validation score is too good to be true. Typical sources: fitting a scaler or imputer on the whole dataset before splitting, selecting features using all labels, duplicate rows across train and test, and features recorded after the outcome (such as "refund issued" when predicting fraud).

The rule: every step that learns from data (scaling, imputing, feature selection, encoding) must be fitted on the training fold only. In scikit-learn, put those steps in a Pipeline so cross-validation refits them inside each fold.

The demonstration below uses data where there is genuinely nothing to learn: 1,000 random features and random labels. A correct evaluation must score around 50%.

Python Data leakage inflates cross-validation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The leaky score is far above 50% on a problem that has no signal, because the feature selection step already "saw" the labels of the rows it was later tested on. The honest pipeline lands near chance (with only 60 rows it can fall a little below 50%). In an interview, the sign to mention is: a validation score that is much better than the production score, or surprisingly good for the problem, is a leakage suspect.

How does gradient descent work, and why does the learning rate matter?

Short answer: Gradient descent minimizes a loss by repeatedly stepping against the gradient: w โ† w โˆ’ ฮท ยท โˆ‚L/โˆ‚w, where ฮท is the learning rate. Too small a rate converges painfully slowly; too large overshoots the minimum and can diverge. Batch gradient descent uses all rows per step, stochastic uses one, and mini-batch uses a small group (the usual choice: noisy enough to escape shallow minima, vectorized enough to be fast). Feature scaling matters because unscaled features make the loss surface long and narrow, so steps zig-zag. Adam, momentum and learning-rate schedules are the usual refinements.

Python Gradient descent and the learning rate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With 0.01 the weight creeps toward 3 and is still at 2.18 after eight steps. With 0.1 it overshoots, oscillates and settles near 3. With 0.3 every step overshoots by more than the last and the weight explodes. Being able to explain these three rows is better than reciting the formula.

What is the difference between bagging and boosting?

Short answer: Both combine many weak models, but in opposite ways.

  • Bagging (bootstrap aggregating) trains many models independently on random resamples of the data and averages their predictions. It mainly reduces variance. A random forest is bagging of decision trees with random feature subsets at each split.
  • Boosting trains models sequentially, each one focusing on the mistakes of the previous ones, and adds them up. It mainly reduces bias (and can overfit if it runs too long). Gradient boosting (XGBoost, LightGBM) fits each new tree to the residual errors.

A single unpruned tree memorizes its training data (100% training accuracy) but generalizes poorly, which is the textbook high-variance model that ensembles fix.

Python One tree, bagging and boosting under cross-validation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The gain from the ensembles here is modest (a couple of points on noisy data), which is normal and worth saying honestly. The single tree scores a perfect 1.0 on the data it was trained on, yet only about 0.81 in validation: a clear sign of overfitting.

Quick answers for the follow-ups

  • Supervised versus unsupervised: supervised learns from labelled examples (classification, regression); unsupervised finds structure without labels (clustering, dimensionality reduction).
  • Normalization versus standardization: normalization rescales to a range such as 0 to 1; standardization subtracts the mean and divides by the standard deviation. Tree models do not need either; distance-based and gradient-based models do.
  • Class imbalance: use stratified splits, class weights or resampling, and judge by precision, recall or PR curves, not accuracy.
  • Curse of dimensionality: as features grow, data becomes sparse and distances lose meaning, so models need more data or fewer features.
  • How to handle overfitting: more data, regularization, simpler model, early stopping, dropout (neural networks), cross-validation, fewer features.
  • Parametric versus non-parametric: parametric models have a fixed number of parameters (linear regression); non-parametric models grow with the data (k-nearest neighbours, trees).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which regularization method can set coefficients exactly to zero, performing feature selection?

    The L1 penalty has a corner at zero, so optimization can land exactly on zero. L2 only shrinks weights smoothly.
  2. 2. You fit a StandardScaler on the full dataset and then do a train/test split. What is the problem?

    The scaler learned the mean and variance of the test rows, so information leaked. Fit it on the training set only, ideally inside a Pipeline.
  3. 3. In gradient descent, a learning rate that is too large usually causes:

    Each step jumps past the minimum, and if the jump grows with each step the loss explodes.
  4. 4. Which statement about bagging and boosting is correct?

    Bagging (random forest) averages independent models to cut variance. Boosting fits each model to the previous model's errors.
  5. 5. Your cross-validation accuracy is 95% but accuracy in production is 60%. What is the most likely first suspect?

    A big gap between validation and production usually means leakage, a split that does not mirror real use, or data drift.

Summary

  • Regularization penalizes large weights: L2 shrinks, L1 shrinks and selects (exact zeros); scale features first and tune the strength by cross-validation.
  • Validate with train, validation and test sets or k-fold cross-validation; use stratified folds for classes and time-ordered splits for time series.
  • Leakage comes from fitting any learning step on data that includes the test rows; use a Pipeline, and distrust scores that look too good.
  • Gradient descent: step against the gradient; too small a learning rate is slow, too large diverges; scale the features.
  • Bagging averages independent models to cut variance; boosting builds models in sequence to cut bias.

Next: Complexity and Data Structure Questions, where the round turns to algorithms.