Skip to content
Tutorials Center
My learning

Topics

Lesson 109 of 111 ยท Practice and Review

Machine Learning Interview Questions

Prepare for machine learning interviews: bias-variance, regularization, metrics, data leakage, trees vs boosting and the curse of dimensionality, each with a short runnable proof.

Machine learning interviews rarely ask you to recite formulas. They ask whether you understand why models fail: why a model that looks perfect on training data is useless in production, why accuracy can be 95% and the model still worthless, why a score can be inflated without a single line of obviously wrong code. The same handful of ideas appears again and again, in different clothes.

This page collects those questions, grouped by theme, with answers short enough to say aloud. Each important claim comes with a tiny experiment you can run right here. Interviewers like candidates who say "let me show you" and then prove it in ten lines, so practice doing exactly that. Related lessons are linked so you can revisit anything that feels shaky.

Six themes behind most questions

Bias and variance Why complexity helps, then hurts.
Regularization Trading a little bias for less variance.
Metrics Which number matches the real cost.
Leakage Information the model should not have.
1 / 2

Bias and variance

1. What are bias and variance?

Bias is error from a model that is too simple to capture the real pattern: a straight line fitted to a curve is wrong in the same way no matter which sample you give it. Variance is error from a model that is too sensitive to the particular sample it saw: a very flexible model fits noise, so a different sample gives a very different model. Total expected error on new data is roughly bias squared plus variance plus irreducible noise that no model can remove. See Overfitting, Underfitting and Bias-Variance.

2. How do you tell overfitting from underfitting using scores?

Compare training and test error. If both are poor and close together, the model underfits (high bias): add features, flexibility or capacity. If training error is excellent and test error is much worse, the model overfits (high variance): add data, simplify, regularize or average several models. The experiment fits polynomials of degree 1, 3 and 15 to noisy points from a sine wave.

Python Underfit, good fit, overfit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Degree 1 is poor on both sets (bias). Degree 15 has the lowest training error but a test error worse than degree 3 (variance). The noise level is 0.3, so the best possible test MSE is about 0.09: degree 3 gets close to it.

3. Why is the bias-variance tradeoff a tradeoff?

Making a model more flexible usually lowers bias and raises variance, and the reverse. The sweet spot is where their sum is smallest. More data shifts the whole picture: variance shrinks, so you can afford a more flexible model. Averaging many high-variance models (bagging, forests) lowers variance without raising bias much, which is why ensembles work so well.

Regularization

4. What does regularization do, and why does it help?

It adds a penalty on coefficient size to the training loss, so the model must justify every large coefficient with a real improvement in fit. That accepts a little bias in exchange for a larger drop in variance. It matters most when you have many features compared with samples, or features that are strongly correlated. See Ridge, Lasso and Elastic Net.

5. How do L1 and L2 differ?

L2 (ridge) penalizes squared coefficients: it shrinks all of them smoothly toward zero but rarely makes one exactly zero. L1 (lasso) penalizes absolute values: it can set coefficients to exactly zero, so it performs feature selection as a side effect. Elastic net mixes both, which helps when correlated features should be kept or dropped together. Always scale features first, otherwise the penalty treats a feature measured in millimeters differently from one in kilometers.

Python OLS, ridge and lasso with many noisy features

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both penalties beat plain least squares on new data. Ridge keeps all 30 coefficients but shrinks them; lasso at alpha=0.1 keeps 9. Push alpha up to 0.3 and lasso keeps only 4, so it starts to throw away real signal and the error rises again: the penalty strength is itself a bias-variance dial.

6. How do you choose the regularization strength?

As a hyperparameter, with cross-validation on the training data (RidgeCV, LassoCV or a grid search), never by looking at the test set. See Hyperparameter Tuning.

Metrics

7. Why can 95% accuracy be a bad result?

With imbalanced classes, a model that always predicts the majority class scores high accuracy while finding nothing. If 5% of transactions are fraud, "never fraud" is 95% accurate and completely useless. Use the confusion matrix, precision, recall, F1 and precision-recall curves, and always compare to a baseline.

Python The accuracy trap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The useless model has 0.95 accuracy and zero recall. The real model barely changes accuracy (0.96) yet catches 60% of positives. The matrix rows are the true classes, columns the predictions.

8. Define precision and recall, and say when you would favor each.

Precision is the share of predicted positives that are truly positive: of the alarms raised, how many were real. Recall is the share of true positives that were found: of the real cases, how many were caught. Favor recall when a miss is expensive (cancer screening, fraud you can review cheaply), and precision when a false alarm is expensive (automatically blocking customers, sending a legal notice). The decision threshold moves you along the tradeoff; see Thresholds, ROC and Precision-Recall Curves.

9. When is ROC AUC misleading?

ROC AUC measures ranking quality across all thresholds and ignores class balance, so with a rare positive class a model can post a high AUC while its precision at any useful threshold is dismal, because false positives are measured against a huge pool of negatives. Average precision (the area under the precision-recall curve) reflects that pain directly.

10. Which regression metric would you choose?

MAE if you want the typical error in the target's own units and outliers should not dominate. RMSE if large errors are disproportionately costly, because squaring punishes them. R squared to compare against the "predict the mean" baseline. Avoid MAPE when the target can be near zero. See Regression Metrics.

Data leakage

11. What is data leakage?

Leakage is any way information the model will not have at prediction time sneaks into training or evaluation, so the measured score is too good. Two common forms: target leakage (a feature that is a consequence of the label, such as "refund issued" when predicting churn) and preprocessing leakage (fitting a scaler, imputer or feature selector on all the data before splitting). Leaky scores look great in the notebook and collapse in production.

12. Show me preprocessing leakage.

Take pure noise: 100 rows, 2,000 random features and random labels. No model can do better than chance, about 0.5. Select the 20 "best" features using all the labels, then cross-validate, and the score looks impressive. Do the same selection inside each training fold and the illusion disappears.

Python Leakage on pure noise

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The leaky estimate is about 0.78 on data that contains no signal at all. The pipeline version is near chance, which is the honest answer. The fix is structural: put every learned preprocessing step in a Pipeline so it is refit on each training fold. For the full checklist, see Nested Cross-Validation and Data Leakage.

13. How do you split data when rows are not independent?

Rows from the same customer, patient or device must stay on the same side of a split, or the model memorizes individuals instead of learning patterns (GroupKFold). For time-ordered data, train on the past and test on the future (TimeSeriesSplit), never a random shuffle. See Cross-Validation for Groups and Time.

14. Why do you need a separate test set if you have cross-validation?

Cross-validation scores guide your choices, so after many comparisons the best one is optimistically biased: you selected it for scoring well. A test set touched once, at the end, gives an unbiased final estimate. Nested cross-validation does the same job when data is too small to hold out a test set.

Trees, forests and boosting

15. Why do single decision trees overfit?

A tree grown until each leaf is pure can memorize any training set, so training accuracy is 100% and predictions swing wildly with small changes in the data: high variance. Limits such as max_depth, min_samples_leaf and pruning trade a little bias for stability. See Controlling Tree Complexity.

16. How do random forests and gradient boosting differ?

A random forest trains many deep trees independently on bootstrap samples with random feature subsets and averages them, which cuts variance. Gradient boosting trains shallow trees one after another, each fitting the errors of the ensemble so far, which cuts bias; it needs a small learning rate and early stopping to avoid overfitting. Forests are robust with little tuning; boosting usually wins when tuned but is more sensitive to its settings.

Python One tree, a forest and boosting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three reach a training accuracy of 1.000 here, but the single tree generalizes worst (0.816 on the test set) while the forest and boosting both reach 0.851. Averaging and boosting fix the variance of one deep tree. See Random Forests and Gradient Boosting.

17. When would you not use a tree-based model?

When you need smooth extrapolation beyond the training range (trees predict flat values outside it), when data is very high dimensional and sparse such as text (linear models excel), or when you need a small, simple, fully explainable formula. For tabular data of moderate size, boosted trees are a strong default.

18. Are impurity-based feature importances trustworthy?

Not always: they are computed on training data and favor features with many distinct values. Permutation importance on held-out data is more reliable, though correlated features can share or hide importance. See Feature Importance and Interpretation.

The curse of dimensionality

19. What is the curse of dimensionality?

As the number of features grows, the volume of the space grows exponentially, so a fixed number of samples becomes extremely sparse, and distances between points concentrate: the nearest and farthest neighbors end up almost equally far away. Methods that rely on distance or local density (k-nearest neighbors, clustering) lose their meaning, and models need exponentially more data to cover the space.

Python Distances concentrate in high dimensions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In 2 dimensions the nearest point is about 50 times closer than the farthest (ratio 0.02). In 1,000 dimensions the ratio is 0.90: "nearest" barely differs from "farthest". See k-Nearest Neighbors.

20. What do you do about it?

Reduce the number of dimensions (feature selection, PCA), use regularization, prefer models that do not depend on raw distance, and collect more data. Domain knowledge that removes irrelevant features is often worth more than any algorithm.

Data preparation questions

What is the difference between standardization and normalization?

Standardization subtracts the mean and divides by the standard deviation, giving mean 0 and standard deviation 1 (StandardScaler). Normalization (min-max scaling) rescales values into a range, usually 0 to 1 (MinMaxScaler). Standardization copes better with outliers and is the default for linear models, SVMs and neural networks; min-max scaling suits bounded inputs such as image pixels. Fit either on the training data only (scaling).

What are upsampling (oversampling) and downsampling (undersampling)?

Both rebalance a skewed dataset. Upsampling adds minority rows, either by duplicating them or by creating synthetic ones (SMOTE). Downsampling removes majority rows, randomly or in a targeted way such as cluster-based undersampling (keep one representative per cluster of majority rows). Upsampling keeps all information but can overfit duplicates and costs time; downsampling is fast but discards data. Apply either only to the training split (imbalanced classes).

How do you impute missing values?

From simplest to richest: the mean, median or mode of the column (the median is robust to outliers); forward or backward fill for ordered data such as time series; a constant such as "unknown" for categories; a neighborhood-based imputer that averages the same column over the most similar rows (KNNImputer); and prediction-based imputers that model each column from the others (IterativeImputer). Add a missing-indicator column when the fact of being missing may carry information (missing values).

How do you handle outliers?

Detect them with the z-score or IQR rules or a model such as an isolation forest, then decide: fix errors, keep real extremes, cap and floor (winsorize) values at chosen percentiles, transform with a log, or use a robust model (outliers).

What is the variance inflation factor (VIF)?

A measure of multicollinearity. For each feature, regress it on all the other features and compute VIF = 1 / (1 - Rยฒ). A VIF of 1 means no overlap, values above 5 to 10 suggest the feature is largely explained by the others, and the coefficients of a linear model become unstable. Drop or combine features, or use ridge regression (multiple regression).

What is the difference between parametric and non-parametric models?

A parametric model assumes a fixed functional form with a fixed number of parameters (linear regression, logistic regression, Naive Bayes). It is compact and needs little data but is biased if the form is wrong. A non-parametric model lets its complexity grow with the data (k-nearest neighbors, decision trees, kernel density estimation). It is flexible, needs more data and stores more (density estimation).

Which assumptions does linear regression make?

Linearity, independent errors, constant variance (homoscedasticity), roughly normal errors and no severe multicollinearity (the assumptions).

Model questions

What is pruning in a decision tree?

Cutting a tree back so that it generalizes. Pre-pruning (early stopping) stops growth with limits such as max_depth, min_samples_split and min_samples_leaf. Post-pruning grows a full tree and then removes branches that do not pay for themselves, for example with cost-complexity pruning (ccp_alpha in scikit-learn) (tree overfitting).

What are the types of Naive Bayes?

Gaussian (continuous features assumed normal within each class), Multinomial (counts, such as word counts), Bernoulli (binary yes/no features) and Complement (a variant of Multinomial for imbalanced text). The choice follows the type of the features (Naive Bayes).

What is the kernel trick?

A kernel computes the dot product of two points as if they had been mapped into a high-dimensional (even infinite-dimensional) space, without ever building that space. Algorithms that only need dot products, such as the SVM, can therefore learn non-linear boundaries at nearly the cost of a linear one (kernel SVMs).

What is bootstrapping, and how does a random forest stay diverse?

A bootstrap sample is drawn from the data with replacement, with the same size as the original. About 63% of the distinct rows appear in each sample, and the other 37% (the out-of-bag rows) give a free validation set. A random forest trains each tree on a different bootstrap sample and lets each split consider only a random subset of the features, so the trees make different mistakes, and averaging different mistakes cancels them (random forests).

What is the difference between ID3, C4.5 and CART?

ID3 handles only categorical features and splits by information gain; C4.5 adds continuous features, missing values, pruning and gain ratio; CART produces binary splits with Gini impurity and handles regression (decision trees).

Which distance would you use for sets or binary vectors?

The Jaccard distance: one minus the size of the intersection divided by the size of the union. Cosine distance suits sparse text vectors, and Hamming distance counts mismatching positions.

Python Bootstrap facts, out-of-bag scores, VIF and Jaccard distance in code

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Unsupervised learning and dimensionality questions

What is K-medoids, and how does it differ from K-means?

K-means uses means (which need not be data points) and squared Euclidean distance; K-medoids uses actual data points as centers and any distance, so it is more robust to outliers but slower (more clustering algorithms).

What is fuzzy c-means?

A clustering method in which each point has a degree of membership in every cluster (the degrees sum to 1) instead of a single label (clustering overview).

What is divisive clustering?

A top-down hierarchical method: start with one cluster, then repeatedly split it (for example with 2-means, called bisecting k-means). Agglomerative clustering works bottom-up.

What is a Dirichlet process mixture model (DPMM)?

A mixture model whose number of components is not fixed in advance. A Dirichlet process prior lets the data decide how many clusters to use, so new components can appear as new data arrives. Scikit-learn's BayesianGaussianMixture with a Dirichlet process prior is a truncated version of it.

Which two methods are called LDA?

Linear Discriminant Analysis (a supervised classifier and dimensionality reducer, discriminant analysis) and Latent Dirichlet Allocation (an unsupervised topic model for text). Always ask which one is meant.

What are Isomap and MDS?

Both are manifold-learning methods that place points in a low-dimensional space while preserving distances. Multidimensional scaling (MDS) preserves straight-line distances; Isomap preserves geodesic distances measured along a neighborhood graph, so it can unroll a curved surface such as a "Swiss roll" (manifold learning).

Time series and drift questions

What does the I in ARIMA stand for, and what is Holt-Winters?

The "I" is integrated: the number of differences needed to make the series stationary. Holt-Winters (triple exponential smoothing) forecasts with a smoothed level, trend and seasonal component (exponential smoothing).

What is concept drift?

The relationship between the inputs and the target changes over time, so a model that was accurate becomes stale. It can be sudden (a new regulation, a competitor's launch), gradual (customer tastes slowly change), incremental (many small steps) or recurring (seasonal patterns that come back). Detect it by monitoring performance and input distributions, and respond by retraining on recent data (monitoring). It differs from data drift, where only the input distribution changes.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A model has 99% training accuracy and 70% test accuracy. What is the most likely problem?

    A large gap between excellent training performance and much worse test performance is the signature of overfitting, that is, high variance.
  2. 2. Which regularizer can set coefficients exactly to zero?

    The absolute-value penalty of L1 creates corners at zero, so lasso drives some coefficients exactly to zero and acts as feature selection. Ridge only shrinks them.
  3. 3. Only 1% of cases are positive. A model that always predicts negative has which property?

    It is right on all negatives (99%) and finds none of the positives, so recall is zero. This is why accuracy alone misleads on imbalanced data.
  4. 4. What is the correct way to select features and estimate performance with cross-validation?

    Any step that learns from labels must be fit only on training data. A Pipeline refits it inside every fold, so the held-out fold never influences the selection.
  5. 5. In very high dimensions, what happens to distances between random points?

    Distances concentrate around a typical value, so the contrast between nearest and farthest shrinks. That weakens distance-based methods such as k-nearest neighbors.

Summary

  • Bias is error from being too simple, variance from being too sensitive; compare train and test scores to diagnose which one you have.
  • Regularization trades a little bias for less variance; L1 selects features, L2 shrinks all of them, and strength is tuned with cross-validation.
  • Pick metrics that match the cost of errors, and distrust accuracy and ROC AUC on imbalanced data.
  • Leakage inflates scores: keep every learned preprocessing step inside a pipeline, split by group or time when rows are related, and touch the test set once.
  • A single deep tree has high variance; forests average it away, boosting reduces bias step by step.
  • In high dimensions, distances concentrate and data gets sparse, so reduce features or use models that do not rely on raw distance.

Next, keep the whole track on one page with the Machine Learning Cheat Sheet.